argparse: a bool indicating if arg was encountered

[8f7af792-b253-47a0-a819-5d9d43b69770]

BPO	44748
Nosy	@rhettinger, @terryjreedy, @Thermi, @🖤Black Joker🖤, @wodny

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = None created_at = labels = ['type-feature', 'library', '3.11'] title = 'argparse: a bool indicating if arg was encountered' updated_at = user = 'https://github.com/Thermi' ``` bugs.python.org fields: ```python activity = actor = 'wodny85' assignee = 'none' closed = False closed_date = None closer = None components = ['Library (Lib)'] creation = creator = 'Thermi' dependencies = [] files = [] hgrepos = [] issue_num = 44748 keywords = [] message_count = 14.0 messages = ['398288', '398351', '398358', '398416', '398417', '398456', '398608', '400957', '400958', '400968', '400969', '400971', '400973', '400986'] nosy_count = 6.0 nosy_names = ['rhettinger', 'terry.reedy', 'paul.j3', 'Thermi', 'joker', 'wodny85'] pr_nums = [] priority = 'normal' resolution = None stage = 'test needed' status = 'open' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue44748' versions = ['Python 3.11'] ```

[8f7af792-b253-47a0-a819-5d9d43b69770]

It'd be great if as part of the namespace returned by argparse.ArgumentParser.parse_args(), there was a bool indicating if a specific argument was encountered.

That could then be used to implement the following behaviour: With a config file loaded as part of the program, overwrite the values loaded from the config file if the argument was encountered in the argument vector.

That's necessary to implement overwriting of settings that were previously set by a different mechanism, e.g. read from a config file.

After all the following order of significance should be used: program defaults \< config file \< argument vector

[586a874d-eaa8-42ea-bb8c-b5611acd5e50]

I would like to use argparse to parse boolean command-line arguments written as "--foo True" or "--foo False". For example:

my_program --my_boolean_flag False

However, the following test code does not do what I would like:

import argparse
parser = argparse.ArgumentParser(description="My parser")
parser.add_argument("--my_bool", type=bool)
cmd_line = ["--my_bool", "False"]
parsed_args = parser.parse(cmd_line)

[8f7af792-b253-47a0-a819-5d9d43b69770]

joker, that is a different issue from the one described here. Please open your own.

[7a064fe6-c535-4d80-a11f-a04ed39056c5]

I've explored something similar in

https://bugs.python.org/issue11588 Add "necessarily inclusive" groups to argparse

There is a local variable in parser._parse_known_args

seen_non_default_actions

that's a set of the actions that have been seen. It is used for testing for required actions, and for mutually_exclusive groups. But making it available to users without altering user API code is awkward.

My latest idea was to add it as an attribute to the parser, or (conditionally) as attribute of the namespace

https://bugs.python.org/issue11588#msg265734

I've also thought about tweaking the interface between

parser._parse_known_args parser.parse_known_args

to do of more of the error checking in the caller, and give the user more opportunity to do their checks. This variable would be part of _parse_known_args output.

Usually though when testing like this comes up on SO, I suggest leaving the defaults as None, and then just using a

     if args.foobar is None:
          # not seen

Defaults are written to the namespace at the start of parsing, and seen arguments overwrite those values (with an added type 'eval' step of remaining defaults at the end).

Keep in mind, though, that the use of subparsers could complicate any of these tweaks.

In reading my posts on https://bugs.python.org/issue26394, I remembered the IPython uses argparse (subclassed) with config. I believe it uses config inputs (default and user) to define the arguments for the parser.

So unless someone comes up with a really clever idea, this is bigger request than it first impressions suggest.

[7a064fe6-c535-4d80-a11f-a04ed39056c5]

Joker

'type=bool' has been discussed in other issues. 'bool' is an existing python function. Only 'bool("")' returns False. Write your own 'type' function if you want to test for specific strings. It's too language-specific to add as a general purpose function.

[7a064fe6-c535-4d80-a11f-a04ed39056c5]

More on the refactoring of error handling in _parse_known_args

https://bugs.python.org/issue29670#msg288990

This is in a issue wanting better handling of the pre-populated "required" arguments,

https://bugs.python.org/issue29670 argparse: does not respect required args pre-populated into namespace

[terryjreedy]

Joker, please don't mess with headers. Enhancements only appear in future versions; argparse is a library module, not a test module.

[rhettinger]

With a config file loaded as part of the program, overwrite the values loaded from the config file if the argument was encountered in the argument vector.

It seems to me that default values can already be used for this purpose:

from argparse import ArgumentParser

config = {'w': 5, 'x': 10, 'y': False, 'z': True}

missing = object()
p = ArgumentParser()
p.add_argument('-x', type=int, default=missing)
p.add_argument('-y', action='store_true', default=missing)
ns = p.parse_args()

# update config for specified values
for parameter, value in vars(ns).items():
    if value is not missing:
        config[parameter] = value

print(config)

[8f7af792-b253-47a0-a819-5d9d43b69770]

Raymond, then you can't show the defaults in the help message.

[rhettinger]

then you can't show the defaults in the help message.

1) The --help option doesn't normally show defaults.

2) Why would you show defaults in help, if you're going to ignore them in favor the values in config whenever they aren't specified. If ignored or overridden, they aren't actually default values.

3) Why not dynamically configure the argparse default values with data from config?

config = {'w': 5, 'x': 10, 'y': False, 'z': True}

p = ArgumentParser()
p.add_argument('-x', type=int, default=config['x'])
p.add_argument('-y', action='store_true', default=config['y'])
ns = p.parse_args(['-h'])

[8f7af792-b253-47a0-a819-5d9d43b69770]

1) True. That'd mean such functionality would not be usable by such a workaround though.

2) ANY setting has a default value. The output in the --help message has to, if any defaults at all are shown, be the same as the actual default values. Storing the default values as part of the argparse.ArgumentParser configuration prevents duplication of the default value declaration in the config file reader, and the argument parser.

What I request is the reverse of what you wrote. I want the order of priority to fall back to the defaults, if no value is specified in the config file. And if an argument is passed via argv, then that value should take precedence over what is set in the config file. This is in the first message in this issue.

3) Two different places to touch when you want to add a new option: 1) Default config declared in program code 2) argparse.ArgumentParser configuration in code.

[7a064fe6-c535-4d80-a11f-a04ed39056c5]

Another way to play with the defaults is to use argparse.SUPPRESS. With such a default, the argument does not appear in the namespace, unless provided by the user.

In [2]: p = argparse.ArgumentParser() ...: p.add_argument('--foo', default=argparse.SUPPRESS, help='foo help') ...: p.add_argument('--bar', default='default') ...: p.add_argument('--baz'); In [3]: args = p.parse_args([]) In [4]: args Out[4]: Namespace(bar='default', baz=None)

Such a namespace can be used to update an existing dict (such as from a config file), changing only keys provided by user (and ones where SUPPRESS does not make sense, such as store_true and positionals).

In [5]: adict = {'foo':'xxx', 'bar':'yyy', 'baz':'zzz'} In [6]: adict.update(vars(args)) In [7]: adict Out[7]: {'foo': 'xxx', 'bar': 'default', 'baz': None}

User provided value:

In [8]: args = p.parse_args(['--foo','foo','--baz','baz']) In [9]: args Out[9]: Namespace(bar='default', baz='baz', foo='foo')

In this code sample I used Ipython. That IDE uses (or at least did some years ago) a custom integration of config and argparse. System default config file(s) set a large number of parameters. Users are encouraged to write their own profile configs (using provided templates). On starting a session, the config is loaded, and used to populate a parser, with arguments, helps and defaults. Thus values are set or reset upto 3 times - default, profile and commandline.

I for example, usually start an ipython session with an alias

alias inumpy3='ipython3 --pylab qt --nosep --term-title --InteractiveShellApp.pylab_import_all=False --TerminalInteractiveShell.xmode=Plain'

Regarding this bug/issue, if someone can come up with a clever tweak that satisfies Thermi, is potentially useful to others, and is clearly backward compatible, great.

But if this issue requires a less-than-ideal-compatible patch, or greater integration of config and argparse, then it needs to be developed as a separate project and tested on PyPi. Also search PyPi; someone may have already done the work.

[rhettinger]

I want the order of priority to fall back to the defaults, if no value is specified in the config file. And if an argument is passed via argv, then that value should take precedence over what is set in the config file.

from collections import ChainMap
from argparse import ArgumentParser

parser = ArgumentParser()
missing = object()
for arg in 'abcde':
    parser.add_argument(f'-{arg}', default=missing)
system_defaults = {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}
config_file =     {'a': 6,         'c': 7,         'e': 8}
command_line = vars(parser.parse_args('-a 8 -b 9'.split()))
command_line = {k: v for k, v in command_line.items() if v is not missing}
combined = ChainMap(command_line, config_file, system_defaults)
print(dict(combined))

This is in the first message in this issue.

The feature request is clear. What problem you're trying to solve isn't clear. What you're looking for is likely some permutation of the above code or setting a argument default to a value in the ChainMap. I think you're ignoring that we already have ways to set default values to anything that is needed and we already have ways to tell is an argument was not encountered (but not both at the same time).

[Paul J3]

So unless someone comes up with a really clever idea, this is bigger request than it first impressions suggest.

I recommend rejecting this feature request. The module is not obliged to be all things to all people. Most variations of the problem already have a solution. We should leave it at that. Extending the namespace with extra boolean arguments would just open a can of worms that would make most users worse off, likely breaking any code that expects the namespace to contain exactly what it already contains.

[cf417b8f-0cb2-4508-ac04-501d63a1e562]

I used a wrapper to default values. This gives me nice help message with ArgumentDefaultsHelpFormatter and easy way to update a config file dictionary with results from parse_args().

from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter, Namespace
from dataclasses import dataclass
from typing import Any

@dataclass
class Default:
    value: Any

    def __str__(self):
        return str(self.value)

    @staticmethod
    def remove_defaults(ns):
        return Namespace(**{ k: v for k, v in ns.__dict__.items() if not isinstance(v, Default)})

    @staticmethod
    def strip_defaults(ns):
        return Namespace(**{ k: v.value if isinstance(v, Default) else v for k, v in ns.__dict__.items() })

p = ArgumentParser(formatter_class=ArgumentDefaultsHelpFormatter)
p.add_argument("--foo", "-f", default=Default(10), help="the foo arg")
p.add_argument("--bar", "-b", default=Default("big-bar"), help="the bar arg")
p.add_argument("--baz", "-z", default=True, help="the baz arg")
options = p.parse_args()
print(options)
print(Default.remove_defaults(options))
print(Default.strip_defaults(options))

$ ./arguments.py -b hello
Namespace(bar='hello', baz=True, foo=Default(value=10))
Namespace(bar='hello', baz=True)
Namespace(bar='hello', baz=True, foo=10)

$ ./arguments.py -b hello -h
usage: arguments.py [-h] [--foo FOO] [--bar BAR] [--baz BAZ]

optional arguments:
  -h, --help         show this help message and exit
  --foo FOO, -f FOO  the foo arg (default: 10)
  --bar BAR, -b BAR  the bar arg (default: big-bar)
  --baz BAZ, -z BAZ  the baz arg (default: True)