Allow multiple `--config`/`-f` options

[raphael-proust]

Currently, invoke only accepts one --config/-f option per invokation.

Multiple config files can be useful to mix configuration from multiple sources. E.g., a general configuration file and then either one for test or one for making more optimisations. leading to two distinct invokations:

invoke -f common.json -f with-tests.json build
invoke -f common.json -f without-tests.json build

These two configurations pass different options to multiple compilers.

---

Workaround: it is possible to manually merge configuration files (i.e., placing the values from common.json into both with-tests.json and without-tests.json). This is unsatisfying because it leads to duplicated information, maintenance costs, etc.

---

Invoke is capable of merging multiple json config files; specifically, as per documentation:

• System-level configuration file stored in /etc/, such as /etc/invoke.yaml.
• User-level configuration file found in the running user’s home directory, e.g. ~/.invoke.yaml.
• Project-level configuration file living next to your top level tasks.py. For example, […] myproject/invoke.yaml.
• Runtime configuration file whose path is given to -f, e.g. inv -f /random/path/to/config_file.yaml.

However, invoke refuses multiple --config/-f options.

---

Solution: merge the files as given with -f in the order that they appear in the command line (i.e., later files overwrite earlier files).

---

How to reproduce:

$ cat tasks.py 
from invoke import task

@task
def build(ctx):
    print('ok')
$ cat a.json 
{
    "a": "a"
}
$ cat b.json 
{
    "b": 0
}
$ invoke -f a.json -f b.json build
No idea what 'b.json' is!

[bitprophet]

So, tl;dr, allow >1 runtime configuration level? Seems reasonable enough at a glance; thanks for the feature request!. I see two obvious ways to accomplish it:

• Perform an initial merge at file load time, such that the resulting Config still only has the one runtime internal data structure. IIRC this would be the easiest option and of course require the least change to how Config works overall. However it feels a bit like it violates the "make it very easy to tell how we went from our config sources to the final value" aspect of keeping levels separated.
• Amend Config's design/behavior so it explicitly keeps an ordered list of runtime structures instead of a single dict and update the merging methodology to account for this. More work, more chance for instability, but arguably cleaner, and helps w/ debugging in the case that the user shot themselves in the foot somewhere between their N runtime configs.

Not sure which I personally like more but I am leaning towards the second option.

[raphael-proust]

Yes, that's the essence of the issue.

I don't know the internals of pyinvoke enough to make any form of decision about how to go about it. As a user of pyinvoke, I'd be satisfied if the list of configuration source was extended by replacing the single -f configuration with multiple of them.

Basically, if I have to think about the series of -f options the same way I think about the series of configuration sources.

Sorry if I don't use the standard vocabulary to describe what I describe. I'm a recent user of pyinvoke.

[bitprophet]

No worries, the internals notes in my comment were mostly notes-to-self :)

Also, just realized this implicitly blocks on #132 - at the moment we don't actually support giving a flag >1 time, but it's something we definitely need to have.

[tedmiston]

I came here after discovering passing multiple runtime configuration files wasn't currently supported as well.

For my use case, as an example I have config files associated with foo objects and also with bar objects.

It would be nice to be able to do something like:

$ inv -f foos/foo1.yaml -f bars/bar1.yaml ...
$ inv -f foos/foo1.yaml -f bars/bar2.yaml ...
$ inv -f foos/foo2.yaml -f bars/bar1.yaml ...
$ inv -f foos/foo2.yaml -f bars/bar2.yaml ...
...

In my case it would be equally sufficient, or even preferable, to be able to do something like:

$ inv --foo=foo1 --bar=bar2 ...

and then have that dynamically load both config files foos/foo1.yaml and bars/bar2.yaml behind the scenes and merge them. I read through the configuration docs, but could not really figure out how to achieve this use case.

For now I'm just going to make one file for each config combination permutation, but with several variations each of 3+ dimensions, it becomes non-ideal fast.

If it's possible to have the config files reference other config files that get pulled in, that would be nice, e.g.:

$ inv -f baz.yaml ...

baz.yaml:

foo: foos/foo1.yaml
bar: bars/bar2.yaml

Please let me know if there is a way to do this that I might be overlooking.

[schperplata]

@bitprophet Would this also apply to other settings like --collection? I would like to keep config in python files (tasks.py), but I would like to have one tasks.py shared across multiple projects (same tasks for multiple projects) and than optionally add tasks in project-specific tasks.py root dir. If --collection argument could be passed multiple times, I would just do:
invoke -c path/to/tasks1.py -c path/to/tasks2.py <taskname>