define discovery of dockmaster.yml files

[briceburg]

Dockmaster is about saving time. Lets allow it to run from anywhere -- no matter the directory you are in -- and be able to quickly target and interact with your compositions/services.

proposal 1 3

dockmaster.yml lookup order

The dm command interacts with a docker-compose composition through a dockmaster.yml "composition configuration file". This file is unique per composition, and its details are described in #2. This proposal covers the process of locating dockmaster.yml.

• Assume the "global configuration file" defines a list of composition_paths (absolute paths to directories containing subdirectories of compositions), e.g.

  /etc/dockmaster/compostions
  |-- accounts.dockerland.org
  |   |-- docker-compose.j2
  |   `-- dockmaster.yml
  |-- machine
  |   |-- docker-compose.yml
  |   |-- Dockerfile-nginx-proxy
  |   |-- dockmaster.yml
  |   `-- nginx-proxy.conf
  `-- registry.dockerland.org
    |-- docker-compose.j2
    `-- dockmaster.yml

• Assume the dm command takes a path as it's first argument, e.g.

  dm <path> <command> [args]
  dm accounts.dockerland.org up
  dm . up
  dm ~/compositions/accounts.dockerland.org
  dm /path/to/accounts.dockerland.org/compostion/

Lookup is performed in the following order

• IF path equals ".";
  • try ./dockmaster.yml
  • fail
• ELSE
  • try path/dockmaster.yml
  • IF path is relative
  • FOR EACH compostion_path in compostion_paths (from global configuration file)
    • try compostion_path/path/dockmaster.yml
  • fail

Example usage

$ dm registry.dockerland.org ps
Connecting to node-c...
             Name                           Command               State            Ports          
-------------------------------------------------------------------------------------------------
registrydockerlandorg_registry_1   /bin/registry /etc/docker/ ...   Up      0.0.0.0:32769->5000/tcp

Notes

• having to specify dm . <command> for current directory is a little awkward, although pure.
• we intend to provide autocompletion for compostions, so you can type dm re<tab> and it will complete dm register.dockerland.org

proposal 2 4

dmc command for managing configuration

Lets introduce a dmc command for configuration management. This command can be used to write to the global configuration (such as managing compostion paths), as well as a the dockmaster.yml compostion configuration file (such as assigning target(s) or registering hooks).

We can cover implementation as a separate issue (#3).

[chrispotter]

I think I'm more in favor of a fallback structure for a dockmaster.yaml as such: 1: cwd/dockmaster.yml 2: DOCKMASTER_PATH/dockmaster.yml environmental variable 3: ~/.dockmaster/configurations.yaml like a .vimrc or something

I guess a DOCKMASTER_FILE=/path/to/my-dockmaster.yml dm blah or a --file flag isn't terrible but it just seems to over complicate the command, but ultimately I can understand the importance of including them... The beauty of dockmaster right now is it's simplicity to run dm configure dm up without having to worry about where or how our compositions are configured or even where these .ymls live.

[chrispotter]

Another thought before I give up for the night, what if we're dynamically building a dockmaster.yaml of some sort.

For instance, I have a dockmaster compose set to be like example 2 as export DOCKMASTER_COMPOSITIONS_PATH=~/compositions so when i run dm list compositions I get something like

accounts.dockerland.org
machine
registry.dockerland.org

but then when i move into a directory that has another composition ~/potter/potterland.blah.com and dm --file= ./ up or something this composition is tracked with the rest of my global compositions, so when i cd any/where/else and run dm list compositions i get

accounts.dockerland.org
machine
registry.dockerland.org
potterland.blah.com

or when I run dm status i could get something like

accounts.dockerland.org       node-a
machine                                 n/a
registry.dockerland.org          node-z
potterland.blah.com              default

or something similar to this. Just a thought to now use dm as your dockingmaster for all active or known docker-compositions within your shell instance or within your current environment. Then when you run dm potterland.blah.com down or unregister it will forget about it until you go back? Or it will just remember forever. This will probably get sticky as you add and remove compositions from your file system. Maybe this is a V2 or beyond scope creep thing

[samtay]

@chrispotter while export DOCKMASTER_COMPOSITIONS_PATH=~/compositions could be added to someone's ~/.shellrc file (and this would be automatically set to the devops-docker repo by BA bootstrap for example) I agree a with point 3. I actually mentioned that to brice offline too. Would like config command similar to git's:

dm config DOCKMASTER_COMPOSITIONS_PATH ~/compositions

that writes to the ~ config file.

But to your second comment, I don't think dm should have any state that isn't explicitly maintained by the user, so I'd vote against something that could change output of the status command after seemingly idempotent commands.

[briceburg]

OK there's some great ideas here! I think we can distill dockmaster even more. Here's my takeaways;

• the idea of a "user dockmaster configuration file is good. lets stick with convention here and lookup this file in home directory before falling back to /etc -- e.g. ~/.dockmaster/dockmaster.cfg || /etc/dockmaster/dockmaster.cfg

• lets allow for more than just one compositions path; e.g. rename DOCKMASTER_COMPOSITIONS_PATH to DOCKMASTER_PATHS or something and express it as a yaml array or ini group in the configuration file (format of this is up-in-the-air)

  [DOCKMASTER_PATHS]
  /etc/dockmaster/compositions
  /home/me/.badevops/git/devops-docker/compositions

• lets limit the dmc command to management of compositions and configuration, and the dm command to interact with compositions. My usage idea for dm is

  dm <path> <command> [args]
  
  # examples
  dm . up
  # => ./dockmaster.yml
  dm accounts.badevops.com up
  # => ./accounts.badevops.com/dockmaster.yml ...
  #      [path1]/accounts.badevops.com/dockmaster.yml ...
  #      [path2]//accounts.badevops.com/dockmaster.yml 

• lets avoid flags in dm. experience shows argparsing is a PITA when you're trying to pass-through args to a command. Most people forget about using -- to terminate argparsing .

I'll followup w/ revised proposals

[briceburg]

OK so I greatly simplified the proposals at the top of this post. We can talk about implementation in #3.

@chrispotter @SamTay @superterran @wrsenn @twslade @Skrath If you're happy with this approach, please give a :+1:

[samtay]

I'm alright with it.. I think it's a bit weird to split all configuration commands into dmc, but I see how that trickles down from avoiding options to dm command and trying to keep it simple as possible, always dm <path> <command>.. I'll continue to think about it but I'm ok with it for now :+1:

[Skrath]

dm <path> <command> is a bit off is you ask me. If you're going to use a command syntax, then you need to start with your command (verb noun format), as this is pretty much how everything else works. If you're going to use a path as a param (and not a flag) then you're aping commands like cp, mv, etc which do not have an additional command parameter, as they are effectively single command programs.

[briceburg]

@Skrath good thought. It is a bit awkward -- but we do need a means of defining which composition to work with, and this does that in a nice an easy and auto-completer friendly way. I'll also note you can think of "path" as "composition".

So we're on the same page;

• Lets say we define define a composition in
  • /etc/dockmaster/compositions/registry.dockerland.org/docker-compose.yml
• and /etc/dockmaster/compostions is listed under _compostionpaths in the "global configuration file".

You can now interact with this compostion anywhere via dm registry.dockerland.org <command>, e.g.

$ cd /where-ever
$ dm registry.dockerland.org  ps
Connecting to node-c...
             Name                           Command               State            Ports          
-------------------------------------------------------------------------------------------------
registrydockerlandorg_registry_1   /bin/registry /etc/docker/ ...   Up      0.0.0.0:32769->5000/tcp

so dm is just a wrapper to docker-compose commands that allows us to extend and define behavior in a dockmaster.yml file [ see #2 ] -- in this case; /etc/dockmaster/compositions/registry.dockerland.org/dockmaster.yml . Notice how the command "Activates Node C" before running the docker-compose ps command.

[Skrath]

Maybe this issue is more that of conception: you're using a globally available command (dm) that requires specific location information. For example, imagine if Git worked like this:

git somereposomewhere.git checkout -b somebranchimade origin/master

And besides looking really weird, this would immediately wreck Git's ability to do the majority of plumbing/piping that it currently relies on, and I can only assume cause a ton of issues with tab completion. (Granted, Git literally cannot function this way since git checkout is actually a single command: git-checkout, but I digress).

The compositions here are effectively location-based contexts that you're applying to a docker-compose command. Bash already has a system for this, it's called CWD. So, instead of having:

[implicit location/context]> [program:dm] [action/command] [additional]

you have:

[nothing]> [program:dm] [explicit location/context] [action/command] [additional]

All for the sake of tab completing the locations? The plan is to ignore already provided contextual information so we can have a globally accessible program that can programatically infer a list of potential contexts merely so that we can then explicitly supply a context, but with tab completion... because using -- is too difficult for users? Or because properly handling command line parameters is hard?

Just... no.

My advice?

• Use proper parameters

or

• Make it interactive

maybe:

• Use some sort of hand-off mechanism to pre-set the context (akin to selecting a db), which I'm pretty sure is how docker-machine works

[briceburg]

I do agree it's not normal to require context as the first argument. This is usually implicit, as in git without the --git-dir flag will traverse upwards until it finds a .git directory and sets the working tree relative to it.

Usually an environment variable and/or a flag is used to set context. E.g. MYSQL_HOST or -h host for MySQL command, DOCKER_HOST for docker command.

So to avoid interaction and retain porcelain execution, I'm reading your suggestion as proper parameters.

It could be that dm implicitly looks in the current directory for dockmasyer.yml, and if and only if the first argument t is -c or --compositon-path we set context relative to that directory. Similar to git --git-dir.

This would allow auto complete because we can make it aware that when the -c flag gets passed, to perform a lookup of compositions instead of docker-compose commands.

So we avoid the awkwardness of 'dm . command', (because . Is weird), but trade in extra characters for typical scenario (e.g. 'dm -c composition up' vs. 'dm composition up').

Basically we go back to the first proposal which is fine.

If we do I produce new flags it gets hairy for users because '--' is unfortunately a hard copy concept.

So the dm usage becomes;

dm [-c|--compositon-path= <path>] [--] command args...

And we can also allow DOCKMASTER_PATH if set, to signal the path to where dockmaster.yml can be found. Basically bringing us back to the first proposal and how dockmaster currently works .

On Thu, Nov 17, 2016, 9:34 PM Chris Rasys notifications@github.com wrote:

Maybe this issue is more that of conception: you're using a globally available command (dm) that requires specific location information. For example, imagine if Git worked like this:

git somereposomewhere.git checkout -b somebranchimade origin/master

And besides looking really weird, this would immediately wreck Git's ability to do the majority of plumbing/piping that it currently relies on, and I can only assume cause a ton of issues with tab completion. (Granted, Git literally cannot function this way since git checkout is actually a single command: git-checkout, but I digress).

The compositions here are effectively location-based contexts that you're applying to a docker-compose command. Bash already has a system for this, it's called CWD. So, instead of having:

[implicit location/context]> [program:dm] [action/command] [additional]

you have:

[nothing]> [program:dm] [explicit location/context] [action/command] [additional]

All for the sake of tab completing the locations? The plan is to ignore already provided contextual information so we can have a globally accessible program that can programatically infer a list of potential contexts merely so that we can then explicitly supply a context, but with tab completion... because using -- is too difficult for users? Or because properly handling command line parameters is hard?

Just... no.

My advice?

• Use proper parameters

or

• Make it interactive

maybe:

• Use some sort of hand-off mechanism to pre-set the context (akin to selecting a db), which I'm pretty sure is how docker-machine works

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/dockerland/dockmaster/issues/1#issuecomment-261433081, or mute the thread https://github.com/notifications/unsubscribe-auth/AAd6oDbdm96NS8F9AQUUbO2W6urASXmqks5q_Q63gaJpZM4K0S0o .

[chrispotter]

I like @Skrath's comparasion to git with it working local to the directory

but also like the idea of hand-off mechanism to pre-set the context service that manages compositions

[briceburg]

That hand off mechanism is more or less setting environment variables. I think we've gone fill circle to first proposal😉

On Thu, Nov 17, 2016, 10:26 PM Chris Potter notifications@github.com wrote:

I like @Skrath https://github.com/Skrath's comparasion to git with like how git working local to the directory

but also like the idea of hand-off mechanism to pre-set the context service that manages compositions

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/dockerland/dockmaster/issues/1#issuecomment-261439845, or mute the thread https://github.com/notifications/unsubscribe-auth/AAd6oKltP6xBBGQ1rSgt7aaxHktRAxX5ks5q_RrngaJpZM4K0S0o .

[briceburg]

closing -- we will document behavior in #5