Config utilities for describing config files and CLI arguments

[NikolausDemmel]

• new module config_utils for a general purpose tool to describe config files, parse yaml config files and expose specific arguments to the command line; includes some unit tests for the config item types
• new module config describes xylem's global configuration; includes the logic to compute sources and cache folders given command line arguments and config files
• new module arguments for global command line arguments (moved from utils); interacts with config to include and process global config arguments
• new module yaml_utils contains customization to loading and dumping yaml (moved (from utils)
• --help argument now uses pydoc.pager to display the help in a less-like scrollable manner on pty's
• help output includes new section config arguments, describing the arguments that correspond to a config file entry
• commands have a new hook to add to the config description; some logic common to all commands was moved to commands.main
• context objects like InstallerContext and SourcesContext now take a config dictionary as argument, which defaults to the global config singleton if not passed explicitly
• high-level API used by the commands (update, resolve, lookup) is also parameterized by an optional config dict argument, which defaults to the global config
• Introduce XylemError as common Exception base class. Having a common base class allows users to catch all xylem related exceptions in one handler.

Output of xylem resolve --help:

usage: xylem resolve [--show-trumped] [--all] [--show-priority]
                     [--show-default-installer] [--os name:version]
                     [--os-features "feature1,feature2,..."]
                     [--core-installers "inst1,inst2,..."]
                     [--use-additional-installers yes|no]
                     [--install-from "inst1:[key1,key2,...] ..."]
                     [--installer-options "inst1:{opt1:val1,...}, ..."]
                     [--user-sources yes|no] [--cache-dir PATH]
                     [--sources-dir PATH] [-h] [--version] [-v] [-d]
                     [--print-config] [--no-user-config]
                     [--prefix XYLEM_PREFIX] [--xylem-dir XYLEM_DIR]
                     [xylem_key [xylem_key ...]]

Lookup a xylem key and resolve to unique, parsed rule based on priorities.

positional arguments:
  xylem_key

optional arguments:
  --show-trumped        Show all possible resolutions for key, also for
                        trumped installers.
  --all                 Resolve all keys with resolution for this OS.
  --show-priority       Show priority of installer.
  --show-default-installer
                        Show installer even if it is the default installer.

config arguments:
  The following typed arguments correspond to entries in the config file.
  Command line argument values are interpreted as YAML and override the
  corresponding entries in user/system config files.

  There are some special cases and short hand notation for parsing config
  arguments:
  * List: the outer '[]' can be omitted; multiple occurrences of the same
  command line argument are concatenated
  * MergingDict: the outer '{}' can be omitted; multiple occurrences of the
  same command line argument are merged; value is merged with the config
  file entries
  * Path: string parsed as-is; path is expanded (e.g. `~`)
  * String: string parsed as-is

  --os name:version     override os detection; if no ':' is present, the
                        entire string is interpreted as the os name and the
                        version is detected
                        (config: `os_override`, type: String)
  --os-features "feature1,feature2,..."
                        OS features to be used. The list of possible values
                        and the default choice is defined by the os plugin for
                        the selected OS.
                        (config: `os_options/features`, type: List)
  --core-installers "inst1,inst2,..."
                        core installers to be used. The default list is
                        defined by the OS plugin for the selected OS.
                        Additional installers as defined by installer plugins
                        may be used on top of the core installers unless the
                        `use_additional_installers` option is set to False.
                        (config: `os_options/installers`, type: List)
  --use-additional-installers yes|no
                        if `True`, use additional installers as defined by
                        installer plugins on top of the core installers
                        (type: Boolean, default: True)
  --install-from "inst1:[key1,key2,...] ..."
                        mapping installer names to list of xylem keys;
                        overwrites the installer priority such that the given
                        keys are only ever installed with the specified
                        installer
                        (type: MergingDict)
  --installer-options "inst1:{opt1:val1,...}, ..."
                        options passed to installer plugins; valid options are
                        specific to each installer plugin
                        (type: MergingDict)
  --user-sources yes|no
                        if `True`, look for sources and cache in user
                        directory instead of system-wide locations
                        (type: Boolean, default: False)
  --cache-dir PATH      override the cache location
                        (type: Path)
  --sources-dir PATH    override the sources directory
                        (type: Path)

global arguments:
  By default xylem operates with system-wide configuration of sources and
  cache. The path under which those are found can be set with the
  XYLEM_PREFIX environment variable (or `--prefix` argument). To instead use
  sources and directories in the user's home directory, make use of the
  `--user- sources` argument. An alternative mode of configuration is by
  setting the XYLEM_DIR environment variable (or `--xylem-dir` argument).
  With that, only config files in that directory are used and sources/cache
  folder is local to the XYLEM_DIR. This can be used by third party tools to
  utilize xylem with config/sources/cache in a temporary directory.

  -h, --help            show this help message and exit
  --version             print xylem version and exit
  -v, --verbose         verbose console output
  -d, --debug           enable debug messages (overwrites the XYLEM_DEBUG
                        environment variable)
  --print-config        print the effective configuration (from command line
                        and config files) and exit
  --no-user-config      disable user config (system-wide config only)
  --prefix XYLEM_PREFIX
                        Set the FHS prefix for finding system-wide
                        configurations and caches. System-wide configuration
                        files will be looked for in 'XYLEM_PREFIX/etc/xylem'
                        and cache files will be placed in
                        'XYLEM_PREFIX/var/cache/xylem'. The default prefix is
                        '/'. The prefix can also be set by the XYLEM_PREFIX
                        environment variable, but the command line option
                        takes precedence.
  --xylem-dir XYLEM_DIR
                        Specify the path of directory to use for configuration
                        and caches. Configuration files will be looked for in
                        'XYLEM_DIR' and caches will be placed in
                        'XYLEM_DIR/cache'. This is intended to be used by
                        tools that want to create a local configuration and
                        cache independent from the system-wide setup. If set,
                        only the configuration files in XYLEM_DIR are used
                        instead of system and user configuration files, i.e.
                        XYLEM_PREFIX will be ignored. This can also be set by
                        the XYLEM_DIR environment variable, but the command
                        line option takes precedence.

Given the following config file in ~/.config/xylem/config.yaml,

install_from:
  homebrew: [foo, bar, baz]
installer_options:
  apt:
    repositories: [{sourceline: "foo bar"}]
os_options:
  features: []
os_override: osx
user_sources: yes

the output of two example commands to illustrate the config functionality follows:

$ xylem resolve boost --os "ubuntu:trusty&python3" --install-from "apt: [foo, bar]" --install-from "pip: [quux, quax]" --print-config
cache_dir: /Users/demmeln/.config/xylem/cache
install_from:
  apt: [foo, bar]
  homebrew: [foo, bar, baz]
  pip: [quux, quax]
installer_options:
  apt:
    repositories: [{sourceline: foo bar}]
os_options:
  features: [python3]
  installers:
os_override: [ubuntu, trusty]
sources_dir: /Users/demmeln/.config/xylem/sources.d
use_additional_installers: true
user_sources: true

$ xylem resolve boost --os "ubuntu:trusty&python3" --install-from "apt: [foo, bar]" --install-from "pip: [quux, quax]" --debug --verbose
overriding OS to [ubuntu:trusty]
initializing database with sources dir `/Users/demmeln/.config/xylem/sources.d` and cache dir `/Users/demmeln/.config/xylem/cache/sources`
No configs found in configured source dir '/Users/demmeln/.config/xylem/sources.d', using default sources.
Loading default source urls
boost --> libboost-all-dev

[NikolausDemmel]

@wjwwood please review.

This has become a bit more extensive than anticipated. More stuff to follow soon.

[wjwwood]

@NikolausDemmel Extensive indeed, I'll do my best to review it tonight.

[wjwwood]

I'm not sure if this is desired, but I get this when using the --print-config global option:

% xylem --print-config
usage: xylem [--os name:version] [--os-features "feature1,feature2,..."]
             [--core-installers "inst1,inst2,..."]
             [--use-additional-installers yes|no]
             [--install-from "inst1:[key1,key2,...] ..."]
             [--installer-options "inst1:{opt1:val1,...}, ..."]
             [--user-sources yes|no] [--cache-dir PATH] [--sources-dir PATH]
             [-h] [--version] [-v] [-d] [--print-config] [--no-user-config]
             [--prefix XYLEM_PREFIX] [--xylem-dir XYLEM_DIR]
             [resolve | lookup | update] ...
xylem: error: too few arguments

[wjwwood]

When I run xylem resolve with no arguments, nothing happens, I would expect usage for resolve:

% xylem resolve

[wjwwood]

Ok, other than the few small comments this is a +1 from me. Please fixup those things I touched on, merge this, and then rebase the other pull requests.

[NikolausDemmel]

I'm not sure if this is desired, but I get this when using the --print-config global option:

It seems to be a limitation of argparse to not allow optional subparsers. The simplest fix might involve http://stackoverflow.com/a/22054398/1813258.

P.S.: I just stumbled upon http://docopt.org/ (the recorded talk is a nice watch). I seems like a really powerful and simple way to do CLI, but unfortunately it looks to be not as practical when options come from different places.

When I run xylem resolve with no arguments, nothing happens

Will fix.

[NikolausDemmel]

Ok, other than the few small comments this is a +1 from me. Please fixup those things I touched on, merge this, and then rebase the other pull requests.

• args/opts is pending a decision
• for xylem --print-config I opened #22

I addressed the other comments as a new commit https://github.com/catkin/xylem/commit/720d3db244c13fbbd1d5595990d8a1e975f4430f ontop of the most recent stuff to avoid rebasing so many commits and possibly manually mergin a bunch of times.

Thanks for the comments.