argparse: provide a simple way to get a programmatically useful list of options

[95eaefa5-eb7e-43a1-8e27-5069868f5d40]

BPO	4256
Nosy	@vstinner, @merwok, @agbuckley, @bitdancer
Files	4256_1.patch 4256_2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', 'library'] title = 'argparse: provide a simple way to get a programmatically useful list of options' updated_at = user = 'https://github.com/agbuckley' ``` bugs.python.org fields: ```python activity = actor = 'vstinner' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'andybuckley' dependencies = [] files = ['16985', '16996'] hgrepos = [] issue_num = 4256 keywords = ['patch'] message_count = 31.0 messages = ['75469', '101989', '102264', '102281', '102306', '102316', '102380', '103586', '103613', '103616', '103643', '103801', '103805', '103825', '103826', '103869', '103876', '103894', '103895', '103897', '103899', '103900', '105421', '105425', '105442', '122605', '153969', '154128', '154186', '154205', '348613'] nosy_count = 12.0 nosy_names = ['bethard', 'vstinner', 'ndim', 'eric.araujo', 'wplappert', 'andybuckley', 'r.david.murray', 'zbysz', 'gruszczy', 'marcs', 'tshepang', 'paul.j3'] pr_nums = [] priority = 'normal' resolution = None stage = 'test needed' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue4256' versions = ['Python 3.5'] ```

[95eaefa5-eb7e-43a1-8e27-5069868f5d40]

optparse is a great option parser, but one thing that would make it even greater would be if it provided a standard option (cf. --help) which lists all the available options in a parseable form. Something prefixed with --help, e.g. --help-options would be ideal since it doesn't clutter the option namespace.

This would provide a simple command-line hook for e.g. bash completion customisation with complete/compgen, which could then easily and maintainably obtain the list of available switches via the --help-options flag rather than hard-coding the option names or attempting to grep the output of --help

It would also be good if the OptionParser provided a simple Python API way to obtain the names of all option switches, rather than having to loop over OptionGroups, calling the unadvertised 'option_list' and 'get_option_name' methods!

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

Are you saying, that for example for this:

gruszczy@gruszczy-laptop:~/Programs/logbuilder$ ./logbuilder --help Usage: logbuilder [options] repo

Options: --version show program's version number and exit -h, --help show this help message and exit -r REGEXP, --regexp=REGEXP filter revisions using regular expression -c CONTAINS, --contains=CONTAINS filter revisions that doesn't contain given string -s START_REV, --start=START_REV first revision to be used in log -e END_REV, --end=END_REV last revision to be used in log -f FILE, --file=FILE file where result will be stored -t TEMPLATE, --template=TEMPLATE template used to display changes -p PURGE, --purge=PURGE remove parts of a commit messages, that match given regexp

you would like to get:

gruszczy@gruszczy-laptop:~/Programs/logbuilder$ ./logbuilder --help-options

--version -h, --help -r, --regexp -c, --contains -s, --start -e, --end -f, --file -t, --template -p, --purge

?

[95eaefa5-eb7e-43a1-8e27-5069868f5d40]

That sort of idea, yes: just a wild thought, but it would be really nice if this was available so that in combination with a standard bash/zsh function, getting basic automatic command completion for scripts built with optparse (and any other implementer of such a scheme) was as simple as adding

complete -F _optparse -o default mycmdname

to the completion script library.

The simple scheme you laid out seems fine to me, but in the best bikeshedding tradition it would be useful to distinguish between options which take an argument and those which don't, pre-empt the need for a format version, and make the parsing even easier by removing cosmetic whitespace, commas etc.:

gruszczy@gruszczy-laptop:~/Programs/logbuilder$ ./logbuilder --help-options

OPTPARSE_FORMAT 0

--version -h --help -r= --regexp= -c= --contains= -s= --start= -e= --end= -f= --file= -t= --template= -p= --purge=

Maybe this is just a pipe-dream, but the need to hand-write basic completion scripts seems so unnecessary, just for lack of any (even de-facto) standardisation. As optparse already enforces/encourages many good habits and conventions, some system like this would further help the integration with shell completion.

Or maybe the existing --help output is good enough for a rather more fiddly standard bash completion parsing function. I've tried writing one of these, but it would hard for it be generally robust since the descriptive strings can contain any structure that they feel like, and could hence mess up the pattern-matching. I'm very happy if someone can out-sed me and make that work!

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

I'll take a look at optparse code and try to provide a patch. But first holidays must finish and I must come back to ma usual residence, where I have programming environment.

[bitdancer]

Please target argparse rather than optparse, or better yet in addition to optparse. And I'm +1 for making it easier to write completion scripts.

[8955c213-fd54-471c-9758-9cc5f49074db]

Someone pointed this out to me earlier:

http://pypi.python.org/pypi/genzshcomp

I believe it's trying to solve the same problem, and claims to work with both optparse and argparse, so it might be worth looking into what it's doing and seeing if there's a useful patch that could be proposed for argparse.

[95eaefa5-eb7e-43a1-8e27-5069868f5d40]

Thanks for the pointers to both of these... I wasn't aware of either. I see argparse has been recently approved for Python stdlib inclusion, too: http://www.python.org/dev/peps/pep-0389/ Congratulations!

As far as I can tell, genzshcomp is parsing the output of the help command to reverse-engineer what the allowed flags should be. Assuming that only one space occurs between the arg and its metavar, this should work 99% or the time... I'm not sure if there is any attempt to be clever when the formatting is ambiguous. But given that the opt parser already contains the structured information, life can be made easier by writing out a more readily parseable format.

Here's an example bash parser function and its usage, for a further-simplified form of the above format where each arg (long or short) gets a line of its own and the arguments are indicated by a separate word as in the current output:

Example input: $ foo --help-options

OPTPARSE_FORMAT 0

--version -h --help -r REGEXP --regexp REGEXP -s N --start N -e M --end M -f FILE --file FILE

and the parser/completion functions:

function _optparse_getargs() { local opts cur prev COMPREPLY=() cur="${COMP_WORDS[COMP_CWORD]}" prev="${COMP_WORDS[COMP_CWORD-1]}"

PREVIFS=$IFS
IFS=$'\n'
for line in `$1 --help-options | egrep '^-'`; do
    opt=`echo $line | sed 's/^\(-[^ ]\+\).*$/\1/'`
    argeq=`echo $line | sed 's/^--[^ ]\+ \([A-Za-z0-9]*\)$/=/'`
    if [[ $argeq != "=" ]]; then argeq=""; fi
    opts="$opts $opt$argeq";
done
IFS=$PREVIFS
unset PREVIFS
opts=`echo $opts | sed -e 's/^ *//' -e 's/ *$//'`

if [[ ${cur} == -* ]] ; then
    COMPREPLY=( $(compgen -W "${opts}" -- ${cur}) )        
    if test -n "$COMPREPLY"; then
        return 0
    fi
fi

    return 0
}

function _foo() { _optparse_getargs foo return 0 }

complete -F _foo -o default foo

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

Patch for optparse with tests. If it's ok, I'll sit down to argparse.

[8955c213-fd54-471c-9758-9cc5f49074db]

Sorry, what does "I'll sit down to" mean? Does that mean you're offering to try to do the argparse patch too? Or that you'd rather someone else do it? (Either one's fine - I just couldn't tell which you meant.)

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

I guess I am using my English too little, that's why I am using polish expressions too often. What I meant was of course, that I will do argparse patch too.

I haven't provided docs for --help-options yet, becuase it is not clear to me, it is an accepted feature. When it is, I will update the patch.

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

Ok, here comes patch for argparse too.

[8955c213-fd54-471c-9758-9cc5f49074db]

Thanks for the patch! One concern I have is that adding --help-options by default has the potential to break existing code, e.g. if someone using optparse or argparse was already defining their own --help-options flag. The backward compatible solution is to have --help-options disabled by default, and ask people to enable it with add_interface=True.

Comments on the argparse patch: I think it's probably overkill to create InterfaceFormatter - just do the appropriate formatting in the _InterfaceAction. I also wouldn't add format_interface or print_interface until someone requests them. Last nit: don't add the takes_value method, just inline your "self.nargs != 0" check in the one place you need it.

[95eaefa5-eb7e-43a1-8e27-5069868f5d40]

The backward compatible solution is to have --help-options disabled by default, and ask people to enable it with add_interface=True.

Or to add the option just before arg parsing, if it has not already been defined?

Thanks for the patches, Filip!

Andy

[bitdancer]

I prefer an approach that allows this option to be defined by default, since if it is not defined by default it defeats part of the purpose of having the option. The program author may not be concerned with completions (or even know about them), but if the option is defined by default then even the programs of those authors can be auto-completed by the generic script.

[bitdancer]

Removing 2.7 since it is now in feature freeze.

[8955c213-fd54-471c-9758-9cc5f49074db]

On Wed, Apr 21, 2010 at 12:36 AM, Andy Buckley \report@bugs.python.org\ wrote:

Or to add the option just before arg parsing, if it has not already been defined?

Something like this was suggested before and it doesn't really work out well. It means the first time you call .parse_args(), your options get modified. So if you do anything with the parser before .parse_args() -- for example, calling .print_help() -- then you don't get the right options.

On Wed, Apr 21, 2010 at 5:12 AM, R. David Murray \report@bugs.python.org\ wrote:

I prefer an approach that allows this option to be defined by default

I agree that it would be best if all command line utilities supported this by default[1]. I'm just not sure how to do it in a backwards compatible way. The fact that the most recent patch against argparse has to modify so many test cases suggests that it's going to have unexpected consequences for a bunch of users.

[1] Though I'd feel more confident in that belief if someone could point me to what the output of other programs that do this looks like so that I could see we were following a standard somewhere.

Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

[merwok]

Hello

Here’s another approach, which has to be used explicitly but provides much more flexible completion: http://pypi.python.org/pypi/optcomplete

I ask the author some time ago if he’d adapt it to argparse; he answered he wouldn’t have time but it shouldn’t be too hard. It’s 221 lines of code (according to sloccount), perhaps worth merging into argparse.

Regards

[bitdancer]

I don't see why --help-options would need to be listed in help. We could pick a more obscure name, too. The point of this option is to support tools, not users.

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

I'll be happy to both fix things pointed by Steven and try some other approach, if that's required, but I would rather do it after a consesus is reached, so I don't have to do the same stuff several times (changing argparse tests requires some work - it's really awesome test suite).

What about optparse? Maybe there we can add this option without much thinking, by checking if help-options is used at the end of parsing? If not, we face the same problem as with argparse.

I don't really understand, why can't we just check if help-options is provided by the user and add our own, if it is not? Even if options are parsed several times (when doing some debugging I have seen my messages printed multiple times in help formatter), we should be able to inspect options in raw form and determine, whether --help-options is used.

[8955c213-fd54-471c-9758-9cc5f49074db]

2010/4/21 Filip Gruszczyński \report@bugs.python.org\:

I don't really understand, why can't we just check if help-options is provided by the user and add our own, if it is not?

I'm sure it would be possible to do it this way. The question is whether it makes sense to from the perspectives of code maintainability and explaining awkward corner cases to users.

On Wed, Apr 21, 2010 at 12:29 PM, R. David Murray wrote:

I don't see why --help-options would need to be listed in help.

Right. For argparse, suppressing the printing of --help-options in the help message is as simple as setting help=SUPPRESS.

We could pick a more obscure name, too.

I think this is probably the best way forward. What is the format that's being printed out? Is this a standard somewhere? Can we name the flag something like "--print-parser-options-in-XXX-format"?

Steve

[merwok]

An obscure name reusing terms like “compword” that can be found easily in Python docs and Bash completion docs would be best.

[8955c213-fd54-471c-9758-9cc5f49074db]

On Wed, Apr 21, 2010 at 12:45 PM, Éric Araujo \report@bugs.python.org\ wrote:

An obscure name reusing terms like “compword” that can be found easily in Python docs and Bash completion docs would be best.

Seems sensible. Does anyone know if zsh uses the same or a different mechanism? If possible, we should be producing output that either of these could use.

Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

[9f70d157-8c5e-4a93-93ff-6125428b3dd1]

So, is there any decision here, so that I could get down to providing better patch?

[8955c213-fd54-471c-9758-9cc5f49074db]

2010/5/9 Filip Gruszczyński \report@bugs.python.org\:

So, is there any decision here, so that I could get down to providing better patch?

I guess I'd like to hear from someone about how these things work in zsh. If we're going to add a hidden flag to *all* parsers, we should really make sure it's compatible/useful for as many of the shells that support this kind of autocomplete as possible...

[bitdancer]

zsh's completion system is completely programmable. I looks like it would be pretty easy to add generic 'python script' support widgets(*) using this hidden option, and probably other neat tricks as well. Something that would make it even more useful for zsh completion would be to include information on the type of the argument when known. A zsh completer could then be programmed to do smart completion on the option value as well.

() You can write a 'widget' and assign it to a key, and then when you use that key the completion widget (shell function) is called and could run the command with the hidden option to get the option info and generate the completion list. That's just the *easiest way to integrate support for this into zsh completion.

[merwok]

Hello Filip. Could you give us a status update on this patch?

[08f81f08-11aa-4faa-851c-2c653ec329f5]

zsh completion is much more powerful. E.g. for git\<SP>log\<SP>\<TAB> I see: completing head \<list-of-heads> completing commit object name completing cached file abspath.c git-lost-found.sh README
aclocal.m4 git-merge-octopus.sh reflog-walk.c
...

git\<SP>\<TAB> completing alias diffab -- alias for 'diff --src-prefix=a/ --dst-prefix=b/' lol -- alias for 'log --graph --decorate --pretty=oneline --abbrev-commit' lola -- alias for 'log --graph --decorate --pretty=oneline --abbrev-commit --all' mergeu -- alias for 'merge --ff-only @{u}' completing main porcelain command add -- add file contents to index am -- apply patches from a mailbox ...

The header parts ('completing commit object name', 'completing head', 'completing cached file') are in bold red. So different completions types are supported, and some help is provided, and completions are split in groups.

Completion for options knows which short/long options go together: git log -\<TAB> prints: ... --oneline -- shorthand for --pretty=oneline --abbrev-commit
--ours -2 -- diff against "our branch" version
--parents -- display parents of commit
--patch -u -p -- generate diff in patch format
...

fish ("a friendly interactive shell" which I don't use but which has some very cool features) prints something like a\%b (Branch) abspath.c (C source code, 4.2kB) abspath.o (Object code, 13kB) aclocal.m4 (M4 macro, 1.4kB) adres (File, 23B) advice.c (C source code, 2.4kB) advice.h (C header, 555B)

I think that for --help-options to be usefull, it should list more information than is needed just for bash completion. At least:

• options, with long and short options specified together
• short help for options (or maybe all of the help)
• option groups if such are used
• metavar names

This last part could be used by the completion script to customize completions for a specific program. E.g. the completion script could know that FILE means a file, and HOST means a host name.

[8955c213-fd54-471c-9758-9cc5f49074db]

So it seems like what bash needs and what zsh needs are pretty different. My feeling so far is that if there isn't "one true format" that argparse can produce and be useful to a wide variety of shells, then it's probably not functionality that belongs in Python core, and instead belongs on PyPI.

So I guess my recommended next step would be to have someone offer help to the maintainer of http://pypi.python.org/pypi/optcomplete to update it to support argparse as well. If and when optcomplete supports argparse, bash, zsh and whatever other common shells people are using, and when optcomplete has significant usage in the field, then we can consider integrating it into the Python stdlib.

[merwok]

zsh completion is much more powerful. I beg to differ :) bash completion can also list more that files, for example only .bz2 files when I complete the bunzip2 command, or Mercurial branch and tag names when I complete hg update, etc. It all depends on the completion script.

I think that there may be enough common ground between the two shells that argparse could print out enough information for both systems. I haven’t read the code of genzshcomp, though.

[08f81f08-11aa-4faa-851c-2c653ec329f5]

ZSH can just present it in a prettier way, and also includes slightly more info (the short explanations, ordering).

> could print out enough information for both systems. Exactly.

ZSH can use bash completion, but then it doesn't display the extra info. It would be nice to keep those optional features in mind to avoid limiting the exported information to that useful for bash.

[vstinner]

This issue is 11 years old has a patch (with 2 versions): it's far from being "newcomer friendly", I remove the "Easy" label.