Option to pass unknown flags to command

[underrun]

related to #623 - but I want the running command to receive rather than ignore unknown flags or return an error.

my use case is loading user defined (in another language) plugin functionality which can accept flags. I can't know these flags at compile time or even at command parsing time because i need the command tell me which plugins to load before i can load it to find out which flags it enables me to accept.

right now my option is to DisableFlagParsing - but that essentially means i need to build my own CLI parser to handle this particular code path and pull out the flags that my command accepts (which cobra could already have done for me).

[FelicianoTech]

This is already built-in via a Unix "standard". Typically, you append a double dash "--" and everything after that can get passed to a subcommand, another binary, etc rather than being parsed by the Cobra command as a flag.

For example:

my-cli command --normal-flag -- --this-is-ignored-by-cobra

[github-actions[bot]]

This issue is being marked as stale due to a long period of inactivity

[underrun]

I hadn't noticed the reply here - sorry.

The bare "--" doesn't work if the goal is to have a single interface where the end user doesn't know what's under the hood. It wouldn't be obvious to the end user what is supposed follow the "--" at all in that case.

The desire is to provide a seamless experience by hanging natively or proxying help info/error into and flags.

On Thu, Apr 9, 2020, 00:05 github-actions[bot] notifications@github.com wrote:

This issue is being marked as stale due to a long period of inactivity

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spf13/cobra/issues/739#issuecomment-611255329, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEOR3YESYXDOCZTDEJLKN3RLUGLFANCNFSM4FSSATVQ .

[jpmcb]

From what I understand of your use case, I believe the double dash -- should work nicely:

POSIX.1-2017

12.2 Utility Syntax Guidelines

Guideline 10:

The first -- argument that is not an option-argument should be accepted as a 
delimiter indicating the end of options. Any following arguments should be 
treated as operands, even if they begin with the '-' character.

From the posix standard

So, each "flag" after the -- can be used as the unknown operand to your application.

The desire is to provide a seamless experience by hanging natively or proxying help info/error into and flags.

This may be achievable from within your application code. For example, within grep:

$ grep -- -v
-v                  # typed as standard input
-v                  # result of grep searching standard input

This will hang waiting for standard input searching for the string -v literal (which is ignored and not passed as an argument to grep itself)

[macintacos]

I'm a bit confused as to how to get the proposed posix-compatible method to work; when I try the above (passing -- and then a flag to be parsed afterwards) I get unknown command: "--this-is-ignored-by-cobra". Can I get some pointers on this? Any examples I can look at?

[jpmcb]

Hi @macintacos - what have you tried? Can you provide a code snippet of what you're seeing? I believe you will still need to parse the options after the double dash -- in your application code if you are attempting to pass in optional arguments

I was able to get this to work with the base cobra init generated application:

$ go run main.go --not-ignored
Error: unknown flag: --not-ignored
...

$ go run main.go --help -- --ignored-by-cobra
A longer description that spans multiple lines and likely contains
examples and usage of using your application. For example:

Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.

Let me know if that is helpful!

[macintacos]

@jpmcb thank you for your quick response!

I think I might've messed up a call to exec.Command, because I shuffled some arguments around and it worked... thanks for at least giving me some hope that it was just something that I was doing wrong and not me running into some weird bug 🙂

Although I think I would agree with the original premise of this issue; it'd be nice if it were possible to tell cobra to blindly accept additional flags/commands if you know what you're doing. In my case, I would take all of the flags and arguments and pass them to exec.Command, since the command I'm creating is essentially running a bash script; it doesn't feel necessary to abstract it in this manner (although I'm fine with the workaround)

[davidovich]

I also think cobra could help here. Something in the lines of python's parse_known_args would allow us to parse known flags and pass other flags to wrapped commands.

[underrun]

@davidovich that's essentially what i was hoping for. again, the -- workaround doesn't work if what you are building is an interface to multiple other underlying things that you want to allow to handle the help/error info themselves.

i completely get that this isn't the common cli use case, but it's so close to doing what is needed to support better delegation that it's frustrating to run into this when it happens.

[jpmcb]

@jharshman @wfernandes thoughts on adding some of this functionality? I'm hesitant since it's baked into the posix standard with the --. And while yes, this may require application developers to include more documentation for their end users and include parsing the unknown args in their application code, I am not sure about subverting standards built into the operating system.

what you are building is an interface to multiple other underlying things ...

I'm also hesitant since the general ethos for unix like commands is to do one thing really well and be enabled to chain in with other commands. For example, instead of a single "line by line" monolithic processor in unix that handles searching, replacing, and cut/pasting lines in files, we have grep, sed, and cut. These chain really well together but also work by themselves. This feels more in line with the spirit of cobra. I don't believe cobra was ever intended to be a tool to generate large, multi-interface applications.

Would it be possible to break apart this monolithic CLI to better handle chaining and different interfaces?

However, I do see the benefit of implementing something like python's parse_known_args. A single interface within cobra to abstract away some of this from the end users would help bring some more complete functionality.

@underrun can you give some examples of what an end user would experience with this "multiple other ulderlying" commands? Maybe I'm having a hard time picturing it.

[davidovich]

I have a use case where I am writing a generic command runner. This runner is configurable by the user, and can call into many utility scripts, but the runner itself cannot know at build time what it will delegate to, only the user knows the flags at invocation time.

A binary plugin system where a driver is responsible for delegating to a subordinate cli is yet another use-case.

I see this kind of usage:

c := &cobra.Command{
    Use:            "run",
    FParseErrWhitelist: cobra.FParseErrWhitelist{UnknownFlags: true},
    RunE: func(cmd *cobra.Command, args []string) error {
        cmd.SilenceUsage = true
        // use the hypothetical UnknownArgs() pflag API
        delegatedArgs := cmd.Flags().UnknownArgs()
        child := exec.Command(args[0], delegatedArgs...)
        return child.Run()
    },
}

[underrun]

This initially came up when working on ksonnet (replaced by tanka) which needed to handle CLI options for managing, using and deploying kubernetes manifests generated from jsonnet (a configuration language) where jsonnet had language specific options we didn't want to replicate because they could change based on the version of jsonnet. We also wanted to maintain a cohesive UX that didn't differentiate between what is was jsonnet and what was ksonnet because the concepts were so very closely related in some cases.

Unix philosophy doesn't preclude doing one conceptual "thing" and doing it well. Git started off as a bunch of individual commands but it makes more conceptual sense as one CLI tool. Sometimes having a CLI for one higher level thing can aide in discoverability of concepts and features, enforcement of best practices, automation of common tasks around that higher level concept and it may be easiest to achieve all that through some composition of lower level tools combined with custom logic.

And I'd love to easily write that custom logic in go :-)

[davidovich]

If anyone wants a workaround, I came up with this atrocity:

func extractUnknownArgs(flags *pflag.FlagSet, args []string) []string {
    unknownArgs := []string{}

    for i := 0; i < len(args); i++ {
        a := args[i]
        var f *pflag.Flag
        if a[0] == '-' {
            if a[1] == '-' {
                f = flags.Lookup(strings.SplitN(a[2:], "=", 2)[0])
            } else {
                for _, s := range a[1:] {
                    f = flags.ShorthandLookup(string(s))
                    if f == nil {
                        break
                    }
                }
            }
        }
        if f != nil {
            if f.NoOptDefVal == "" && i+1 < len(args) && f.Value.String() == args[i+1] {
                i++
            }
            continue
        }
        unknownArgs = append(unknownArgs, a)
    }
    return unknownArgs
}

[freshsun]

I have another user case here. I am developing an assisting tool for hadoop/spark, people use a cobra-developed-cli as a prefix when calling hadoop/spark-submit command, so it can communicate with an custom http server instead of a native one. The logic is implemented within the prefix cobra-cli. Now is the problem, I cannot implement custom flags for cobra-cli, because either I reimplement all hadoop/spark-submit flags again, or let cobra-cli ignore all flags. What i am doing now is using ENV variables as flag, but it is not very friendly, and functional restricted, and help message is also hard to implement.

[jpmcb]

Thanks for the feedback all! Yes, I think it makes sense to implement this. However, I think to have a streamlined, unified interface, the pflags library will need to include a flag set of unknown flags.

Looks like there's already an open PR for this. Maybe we can generate some traction there: https://github.com/spf13/pflag/pull/199

Once we have a pflags unknown flag set, we should be able to easily get it back to the cobra application code.

[davidovich]

Once we have a pflags unknown flag set, we should be able to easily get it back to the cobra application code.

In fact, just a module version bump would suffice as Flags() returns a FlagSet already, on which we can call the new SetUnknownFlagsSlice() API.

[github-actions[bot]]

This issue is being marked as stale due to a long period of inactivity

[davidovich]

Let's keep it active.

[janosmiko]

+1 here. Also @davidovich do you have an idea to a workaround for disabling --help (pass it as well to the subcommand)?

[davidovich]

@mixe3y Unfortunately not, and the help is quite hardcoded currently. I have a need for this too, but haven't come around to implementing that. As a workaround, I would probably scan os.Args and remove --help and -h before parsing with cobra, and then inject the --help to subcommand if if was found. But that does not remove the help from cobra as it insists on having a default help function.

[janosmiko]

@mixe3y Unfortunately not, and the help is quite hardcoded currently. I have a need for this too, but haven't come around to implementing that. As a workaround, I would probably scan os.Args and remove --help and -h before parsing with cobra, and then inject the --help to subcommand if if was found. But that does not remove the help from cobra as it insists on having a default help function.

I was able to do that by setting DisableFlagParsing to true. However this will disable all the flags. :(

[github-actions[bot]]

This issue is being marked as stale due to a long period of inactivity

[ghostsquad]

definitely not stale. I just ran into the same thing... I want to pass some arbitrary things that look like flags to docker

cli docker run alpine /bin/sh -c 'echo "hello"'

Cobra is not interpreting the -c as an arg, but rather as a flag...

[marckhouzam]

@ghostsquad I think your particular scenario would fit well with the use of --: cli docker run alpine -- /bin/sh -c 'echo "hello"'

[marckhouzam]

I agree this enhancement would be useful. For example, the helm binary currently has to have its own flag parser when it handles plugins. Helm doesn't know about each different plugin's flags so it cannot tell Cobra about them, but at the same time, helm has global flags that can be used with any plugins.

It would be nice for Cobra to parse the known global flags and hand over the unknown plugin flags.

[cornfeedhobo]

@davidovich fwiw, my fork has a disable help option, and has support for exposing unknowns, and it works with cobra (unit tests are run with my fork in place).

[SpicyLemon]

This is a feature I would like to see too.

My use case is similar to the others stated: a wrapper for a set of 3rd party utilities. The possible flags/options/arguments cannot be known ahead of time. Converting usage from the other commands to my wrapper should be as seamless as possible. In some cases, it'll just be replacing the executed command; in almost all other cases, it'll be replacing the command and adding one flag.

My wrapper will have its own flags/options (and possibly config via Viper) that I'd like to be able to mix in with the other programs' args flags and options. But any unknown flags/options should just be provided directly as entries in the args []string that is provided to the cobra command's runner.

In the event my wrapper defines a flag that is meant for the other program, it makes sense to use -- to separate them, but it'd be annoying to always have to provide it. I want the replacement of the other programs with my wrapper to be as seamless and native as possible. But -- should rarely be needed, and definitely shouldn't be required.

[philregier]

To pot a finer point on the warewulf issue, I wanted to point out that it's been commented there that when a command is going to be passed through two tools or modules which each use Cobra, it becomes necessary to specify -- twice.

By extension, for those stumbling across this issue with toolchains and frameworks which you might not know intimately (as in my own case), you may have to try first --, then -- --, then perhaps -- -- --, and so on. Better yet, different invocations may require different numbers of -- even for the same application; if I understand the progression of this issue correctly, it is up to the "user" to determine the number of --s required on a case-by-case basis.

[DanThePutzer]

I'm honestly a bit dazzled that this isn't supported out of the box. Took this for a no-brainer and was looking through the docs to find out how to pass some flags to Terraform under the hood. Never even considered this isn't supported :eyes:

I'm assuming quite a few people feel the same way when they land here after searching for a while

[marckhouzam]

@DanThePutzer to be precise, Terraform does not use Cobra (none of the Hashicorp CLI seem to). It does use spf13/pflag however, so you may be interested in the corresponding discussion https://github.com/spf13/pflag/pull/199

[DanThePutzer]

@marckhouzam apologies I might've formulated it a bit weird. What I meant is my CLI tool (https://github.com/Pluralith/pluralith-cli) runs Terraform under the hood and I would like to pass all unknown args my CLI gets on to terraform under the hood. Hope it makes more sense now

[zquintana]

Any movement on this? +1

[seyedmmousavi]

Thank's a lot for awesome 🐍!
In my case, I implementing a custom mini Data Migration tool to make steps carefully planned, automated and safe (+tests). So, this tool incorporates several CLIs.
I need to make it possible to pass any extra arguments to the callee CLI in order to handle some minor situations instead of implementing many minor flags for all called CLIs.
The -- solves this but is not nice.

[cornfeedhobo]

The way pflag parses flags, it's not easy to distinguish an unknown flag vs a flag value or argument. My fork has an attempt to remedy this, and I know of companies currently using it in production, but it too suffers from a few edge cases because pflag would need to be completely re-written to support this use case.

[wagoodman]

For folks looking for a workaround in certain circumstances, here's what I did:

• split os.Args on -- yourself, you should get 1 to 2 lists
• pass the first list to the root command via cmd.SetArgs()
• pass the second list to your run function for processing ignored args

This doesn't solve having the unknown args being parsed in some way, but it does allow for an application that is trying to be a pass through for another to still have custom flags itself (instead of disabling flag parsing altogether).

func splitArgs(args []string) ([]string, []string) {
    if len(args) == 0 || len(args) == 1 {
        return nil, nil
    }

    args = args[1:]

    for i, arg := range args {
        if arg == "--" {
            return args[:i], args[i+1:]
        }
    }
    return args, nil
}

func setupCommands() *cobra.Command {
  root := &cobra.Command{}
  args, ignoredArgs := splitArgs(os.Args)
  root.SetArgs(args)

  // do what you want with ignoredArgs!.. maybe make it available for another command run function

  return root
}

[M09Ic]

In the following example:

cli docker run alpine /bin/sh -c 'echo "hello"'

In this case, all arguments after the first positional argument should be treated as positional arguments, not as flags.

If docker run is a subcommand, then everything after alpine should be treated as arguments for the subcommand. Perhaps there should be a configuration option to allow this behavior.

For instance, if both the CLI and subcommands share similar flags:

cli -t 10 docker run -t ... alpine /bin/bash -t ...

The first -t should be parsed as a flag for cli, the second -t should be parsed as a flag for the subcommand docker run, and the third -t, because it comes after the positional argument alpine, should automatically be treated as a positional argument rather than as a flag.

I’m not sure if my description is entirely clear, but this issue seems broader than just handling unknownFlags.