rework cli options and environment variables

[smola]

• Remove dependency from core-ŕetrieval container. Now there are no hidden options, all of them are explicit and listed in --help. Note that with our current usage, container/config is rarely used outside of the scope of the cli interface, since that's where everything is instantiated and dependencies are injected, "library code" outside cli should not magically instantiate these.

• Add environment variables to all configuration options. Every setting has a cli flag and an environment variable. One-off arguments (e.g. for file and pack subcommands) are now positional arguments instead of flags.

• Prefix all environment variables with BORGES_.

• Add long descriptions for commands.

• Homogeneize some equivalent options that had different names in different subcommands. For example, pack now uses --root-repositories-dir instead of output. Note that this means that this CLI is not backwards compatible.

• Pack now shares underlying initialization code with consumer, so it should support HDFS, bucket size, etc, instead of having those settings hardcoded.

[smola]

The juice is in the first commit.

Example of the result:

Usage:
  borges [OPTIONS] consumer [consumer-OPTIONS]

This consumer fetches, packs and stores repositories. It reads one job per repository. Jobs should be produced witht he producer command.

Help Options:
  -h, --help                            Show this help message

[consumer command options]
          --queue=                      queue name (default: borges) [$BORGES_QUEUE]
          --broker=                     broker service URI (default: amqp://localhost:5672) [$BORGES_BROKER]
          --locking=                    locking service configuration (default: local:) [$BORGES_LOCKING]
          --workers=                    number of workers (default: 8) [$BORGES_WORKERS]
          --timeout=                    deadline to process a job (default: 10h) [$BORGES_TIMEOUT]
          --root-repositories-dir=      path to the directory storing rooted repositories (can be local path or hdfs://) (default: /tmp/root-repositories)
                                        [$BORGES_ROOT_REPOSITORIES_DIR]
          --root-repositories-temp-dir=
          --bucket-size=                if higher than zero, repositories are stored in bucket directories with a prefix of the given amount of characters
                                        from its root hash (default: 0) [$BORGES_BUCKET_SIZE]
          --temp-dir=                   path of temporary directory to clone repositories into (default: /tmp/borges) [$BORGES_TEMP_DIR]
          --temp-dir-clean              clean temporary directory before starting [$BORGES_TEMP_DIR_CLEAN]
          --database-driver=            database driver (default: postgres) [$BORGES_DATABASE_DRIVER]
          --database=                   database connection string (default:
                                        postgres://testing:testing@0.0.0.0:5432/testing?application_name=borges&sslmode=disable&connect_timeout=30)
                                        [$BORGES_DATABASE]

    debug options:
          --metrics                     expose a metrics endpoint using an HTTP server [$BORGES_METRICS_ENABLE]
          --metrics-port=               port to bind metrics to (default: 6062) [$BORGES_METRICS_PORT]
          --profiler                    start CPU, memory and block profilers [$BORGES_PROFILER_ENABLE]
          --profiler-port=              port to bind profiler to (default: 6061) [$BORGES_PROFILER_PORT]

[jfontan]

I believe we don't use --root-repositories-temp-dir anymore. I'll take a look and do a PR to core-retrieval if this is the case.

Also I would change --root-repositories-dir to --output in both commands. It's shorter, easier to understand and can have a short option -o.

[smola]

@jfontan I think --root-repositories-temp-dir should be removed if possible, although that implies some behaviour change I was not sure if I should do though. Note that I did not include --hdfs, and rely on a hdfs:// URL being passed to --root-repositories.

The problem I see with --output is that it is not purely an output directory, since it can also be read on updates.

Also note that you can setup Bash completion for an improved user experience:

_completion_example() {
    # All arguments except the first one
    args=("${COMP_WORDS[@]:1:$COMP_CWORD}")

    # Only split on newlines
    local IFS=$'\n'

    # Call completion (note that the first element of COMP_WORDS is
    # the executable itself)
    COMPREPLY=($(GO_FLAGS_COMPLETION=1 ${COMP_WORDS[0]} "${args[@]}"))
    return 0
}

complete -F _completion_example completion-example

See https://godoc.org/github.com/jessevdk/go-flags#hdr-Completion

[jfontan]

--root-repositories-temp-dir is no longer used. See src-d/core-retrieval#65

[smola]

@mcuadros What do you think about this approach for both flags and environment variables?

[smola]

I thought this would expose every possible setting in --help, but turns out that logging (via go-log) and queue (via go-queue) are still hidden (thanks @jfontan for pointing this out). I would like to add these explicitly to --help too.

[smola]

Closing in favor of https://github.com/src-d/borges/pull/338#issuecomment-419116618