feat(ffyaml): add WithKeyPath parser option

[pkazmier]

Specifying a key path points the parser to a specific section of the YAML file. For example, given the YAML config:

config:
  dev:
    value: 10
  prod:
    value: 100

The following will use the "dev" portion of the configuration file:

ff.Parse(fs, []string{},
    ff.WithConfigFile(file),
    ff.WithConfigFileParser(
        ffyaml.New(ffyaml.WithKeyPath("config", "dev")).Parse
    ),
)

Use cases for the feature include:

• Placing different configurations in the same file
• Using the same file for multiple tools (besides this parser)
• Using the same file for different instances of this parser

[peterbourgon]

I believe this is out-of-scope for package ff.

If you want to have a top-level config file like you say, e.g.

config:
  dev:
    value: 10
  prod:
    value: 100

That's fine. But you definitely shouldn't deploy that file directly to both dev and prod hosts. You should deploy a transformation of that file, which is environment-specific. For example, on dev hosts you would deploy

value: 10

and on prod hosts you would deploy

value: 100

Environment, e.g. dev vs. prod, is implicit at runtime, it's not something that a program should know about directly.

Specifically, this doesn't work:

ff.Parse(fs, []string{},
    ff.WithConfigFile(file),
    ff.WithConfigFileParser(
        ffyaml.New(ffyaml.WithKeyPath("config", "dev")).Parse
    ),
)

That's because a program, in general, should have no way of knowing that it should pass "dev" as a parameter to WithKeyPath. Dev vs. prod is something established by the configuration passed to the program, not a decision the program itself makes.

The config file used by this package is expected to be scoped to the specific context, environment, etc. of the running application. Consequently, all of your use cases — placing different configurations in the same file, using the same file for multiple tools (besides this parser), and using the same file for different instances of this parser — seem to me to be errors of design.

But I could be overlooking something. Can you give more detail on those use cases, which might suggest otherwise?

[pkazmier]

Perhaps my examples were not the best. Let me share my actual use case. I am building a CLI tool that takes one or more user-defined commands (cmd1, cmd2, cmd3 in example below)—each with their own options and flag sets. These commands are not subcommands of each other, but rather equal top-level commands.

$ cli cmd1 -opt1a 10 -opt1b 20 -opt1c 30 cmd2 -opt2a 40 cmd3 -opt3a 50 -opt3b 60

Users of the cli program should be able to specify all of their default configurations for all of the possible used-defined commands in a single YAML file:

config:
  cmd1:
    opt1a: 10
    opt1b: 20
    opt1c: 30
  cmd2:
    opt2a: 40
  cmd3:
    opt3a: 50
    opt3b: 60

So, my thought was to split cli's argv by commands and then call ff.Parse on each of them separately with a YAML configuration parser that specifies the correct key path for that section of the configuration file. This is why I added the WithKeyPath for the YAML configuration parser.

I thought I'd contribute back to your wonderful project since I thought it was useful. If you don't believe this is a fit for your project, I'll simply maintain my own YAML configuration parser.

[peterbourgon]

What is the relationship of cli and a subcommand like cmd1? How is cmd1 defined/expressed to cli?

edit

cli cmd1 -opt1a 10 -opt1b 20 -opt1c 30 cmd2 -opt2a 40 cmd3 -opt3a 50 -opt3b 60

This command is definitely not valid. If you want to do this, it's 3 separate commands:

1. cli cmd1 -opt1a=10 -opt1b=20 -opt1c=30
2. cli cmd2 -opt2a=40
3. cli cmd3 -opt3a=50 -opt3b=60

[pkazmier]

cli is a generic tool that runs one or more user-defined commands (statically compiled plug-ins) against a list of one or more items concurrently in a pipeline like manner. As a concrete example, here is how one would use this tool to list the VPCs of every prod AWS account using SAML to obtain credentials:

$ cli metafilter -include env=prod awssaml -user USER -role ROLE listvpcs -region us-east-1

Each command (metafilter, awssaml, listvpcs) above implements in interface to allow them to be used with my cli tool. The commands each implement lifecycle methods where the output of a prior command can modify the item list passed to subsequent commands. Further, each command can register itself as a service, so downstream commands can use them.

I simply need a way to configure each command. Most CLI libraries only support subcommands, but I need support for multiple top-level commands. Each top-level command can specify its own flag set independent of the others. By splitting on the commands (haven't figured how I'll do that yet, maybe use a delimiter or a common prefix for commands or split on known command names), I can then invoke ff.Parse for each of them. And with the WithKeyPath, allow users to configure each of these independent user-defined commands in the same config file instead of having individual config files for each, which is unwieldy given the library of commands there are (50+).

[peterbourgon]

I simply need a way to configure each command. Most CLI libraries only support subcommands, but I need support for multiple top-level commands. Each top-level command can specify its own flag set independent of the others. By splitting on the commands (haven't figured how I'll do that yet, maybe use a delimiter or a common prefix for commands or split on known command names), I can then invoke ff.Parse for each of them. And with the WithKeyPath, allow users to configure each of these independent user-defined commands in the same config file instead of having individual config files for each, which is unwieldy given the library of commands there are (50+).

This is an unsound design. Each invocation of a ffcli binary is a single traversal of the command tree, from the root command to a single specific downstream subcommand. There is no concept of multiple concurrent top-level commands in the sense that you mean here. If you want to pipeline (chain) multiple commands together, that's done via the shell, not the command parser library.

# no
$ cli metafilter --include="env=prod" awssaml --user=USER --role=ROLE listvpcs --region=us-east-1

# yes
$ cli metafilter --include="env=prod" | cli awssaml --user=USER --role=ROLE | cli listvpcs --region=us-east-1

[pkazmier]

I understand that ffcli has no concept of multiple top-level commands, but the ff package is flexible enough for me to achieve what I desire—a means to allow plug-in developers to specify their own flags independently, but allowing users to use a single configuration file.

Shell pipelines work well when passing strings between programs, but my commands don't just pass data via a pipeline. They also register themselves in a Context, so downstream commands can invoke services that they may provide. For example, the "metafilter" command exposes a MetaForItem service that the "listvpc" author might invoke to get additional data on a particular item. Likewise, the "awssaml" command exposes a AwsCredentials method to get tokens for AWS accounts. This allows the "listvpc" author to use those services if needed by getting a reference to the plug-in via the Context.

So, no, one cannot use a shell pipeline.

I feel we've strayed far from the simple addition I made to the YAML configuration parser. I only shared as I like to contribute back, but appreciate that you do not want to include. No worries or offense taken. I simply would like to say thank you for your amazing library and its extensibility that has allowed me to solve my problem.

[pkazmier]

For more context on the use case I was trying to explain, it's really about middleware and allowing a user to specify one or more middlewares for the command they want to run:

$ cli mw1 --role==ROLE mw2 --include ACTIVE cmd

Where the middlewares wrap a Command interface implemented by the cmd. So, I just was trying to find a way to allow the middleware authors to specify their own flag sets and use the same cli configuration file—thus the reason for the WithKeyPath.

[peterbourgon]

Thanks for that context, it answers a lot of my questions.

ffcli definitely doesn't support any concept of middleware as you describe in this issue. Each invocation of a ffcli root command is expected to resolve to a single, linear traversal of the command tree. cli mw1 [...] mw2 [...] cmd expects mw1 to be a subcommand of the root command, mw2 to be a subcommand of mw1, and cmd to be a subcommand of mw2.

This is a fundamental assumption of the design. Commands aren't interfaces that can be decorated. They're concrete things, defined as Exec functions, with access to a (single) flag set, and a (single) list of user args.

If running cmd may result in different behavior based on --role=ROLE or --include=ACTIVE, then role and active would need to be explicitly understood, parsed, etc. by cmd's Exec function. The objectctl example provides a (simple) example of how to extend knowledge of higher-level flags to lower-level subcommands.

edit: If mw1 and mw2 are essentially just additional flags that apply to specific commands, then the way to model that would be to define mw1 and mw2 as flagsets which the relevant commands (e.g. cmd) would receive during construction, and include in the flagset it constructed for itself.