Type hinting ("typing" module)

[jacobsvante]

First of all, THANKS @neithere for this awesome project, it's never been easier to set up a command line tool!

Just wanted to check on you to see how you feel about supporting the new typing module? This way it would be possible to specify type on positional arguments without using the @argh.arg decorator.. This would make argh even more great i.m.o.

[neithere]

Hi Jacob,

Thank you for your interest and for the idea. I agree that we should support typing. Originally there was no convention on the usage of annotations, so I chose to tuck argument documentation there. It allowed to get rid of the @arg decorator when it was used solely for the help. However, as annotations now seem to be used mostly with typing and it would be at least useful as the docs are, we should change the way argh interprets annotations.

Possible approach:

• deprecate textual annotations;
• introspect the annotation and show deprecation warning if it's a string;
• remove old behaviour in some later version.

Obstacles:

• @arg('x', help='blah') will have to be added in certain cases;
• it's not clear how exactly we would use types.Generic e.g. for string coercion.

[jacobsvante]

Thanks for your input @neithere. I did not think of the annotation functionality of Python 3 and tbh I've never used it. Is there a way to combine both? I like the ability to idea of inlining the documentation also 😅.

[neithere]

So, the library began its life when I had some free time in an airport. Today I had a similar time slot in the same place, so I decided to give argh+typing a try :) Unfortunately, when I started writing unit tests, I couldn't come up with anything meaningful. I kinda feel that it should make sense, but how? ))

Do you have any hypothetical example that we could put into unit tests and try to make it work?

Andy

[cottrell]

Where did this land in the last few years? I might have a look unless someone would suggest another library that does similar things? I have been quite happy with argh over the time I've been using it so might just try to kick this along. A typed example might be something like this:

def f(*, a: int, b: str, c: float):
    pass

[presheaf]

I also found myself wanting this. Currently, if you have a function you want to argh.dispatch_command on with some arguments you want to type, in the absence of default values argh can use to infer types, you're forced to do something like the following for x, y, z to end up being ints.

@argh.arg('x', type=int)
@argh.arg('y', type=int)
@argh.arg('z', type=int)
def myfunc(x, y, z):
    ...

It would be a lot nicer IMO if you could just write

def myfunc(x: int, y: int, z: int):
    ...

This would take the place of th documentation-type annotations argh currently supports. I can't think of a clean way to make them go together which wouldn't abuse the type hinting system, Maybe those are a better fit for a decorator?

@argh.doc({
    'x': 'information about x',
    'z': 'information about z'
})
def myfunc(x: int, y: int, z: int):
    ...

I think I would be up for implementing this, if there is interest.

[neithere]

So... This is definitely something that needs to go into Argh 1.0. Will bring the library to a whole new level of DRYness that was not technically achievable at the time when its first versions were created.

Issue #144 adds more ideas which I like a lot: using Optional and List/Tuple to infer more than just type.

So e.g. for func(foo: Optional[List[str]]: None) we would typing.get_args(hint) for foo and:

• descend into the first nested item, do typing.get_origin(x), discover that it's a list → set flag "multi-value"
• descend into the second nested item, discover that it's NoneType → set flag "optional"
• set nargs to * because it's "multi-value" and "optional".

Mapping the typing hints to argparse argument declarations is not an easy task. There will be dubious cases.

I think we need to start by drafting a list of possible mappings, from simple and obvious ones to complicated ones (with Union and so on).

For example:

• foo: 'hello' → add_argument('foo', help='hello')
  • the original "hack" to document arguments via annotations, should be deprecated as it is considered an error by mypy.
• foo: str → add_argument('foo', help='str')
• foo: Optional[str]: None → add_argument('foo', type=str, default=None, help='Optional[str]') (I'll omit help in the rest)
• foo: List[str] → add_argument('foo', nargs='+')
• foo: Optional(List[str]) → add_argument('foo', nargs='*')
• foo: Dict[str, str] → ?
  • I can imagine parsing JSON or something in that vein but it's "too smart" for default behaviour; could be enabled via some decorator like @argh.hints(parse_json_for=[Dict, List]) or @argh.plugin('json').enable_for_args('foo', 'bar').

[presheaf]

Here are some suggestions for two somewhat complex but (I think) useful type hints.

• foo: Literal['one', 'two', 'three'] -> add_argument('foo', choices=['one', 'two', 'three'] (it would be very cool if argument completion could also hook into this)
• foo: Tuple[str, int, float] -> add_argument('foo', nargs=3), followed by (e.g.) args = parser.parse_args(); args.foo[1] = int(args.foo[1]); args.foo[2] = float(args.foo[2])

I'm a little unsure about what I'd expect to happen for foo: UnsupportedType. UnsupportedType(passed_arg) seems a sane default, but sometimes won't be what is expected, so probably some care should be taken with informative error messages and where conversion functions can be passed via decorator. Provided common ones like JSON parsing are easily available, I don't think this will be too annoying.

[neithere]

Just a note: this can get tricky but also very interesting and useful if/when we decide to support overloading. Definitely not part of MVP though.

[neithere]

FYI, I'm doing a significant revamp as part of #191 and keeping this in mind as the next step.

Approximate roadmap, incremental and realistic:

1. Get rid of the old annotations.
   • deprecated in v0.28
   • to be removed in the upcoming release (ETA: mid-Oct 2023).
2. Introduce very basic typing-based guessing and deprecate the old one (based on defaults and choices).
   • probably v0.31 (Nov/Dec 2023).
3. Get rid of the old guessing, extend typing-based guessing (still keep it simple).
   • probably v1.0 (EOY 2023 / early 2024).
4. Continue extending the typing-based guessing and find a way to keep it DRY for help.

[neithere]

Perhaps one of the best ways to replace @arg would be the Annotated type (PEP-593, included since Python 3.9), basically the standard way to add metadata to type hints.

Usage example:

def load_dump(
    path: Annotated[str, argh.Help("path to the dump file to load")],
    format: Annotated[str, argh.Choices(FORMAT_CHOICES), argh.Help("dump file format")] = DEFAULT_CHOICE
) -> str:
    ...

Accessed during parser assembly via func.__annotations__["some_arg"].__metadata__ (a tuple of metadata items).

[neithere]

139 is a useful edge case example: list[str] with nargs="+".

[neithere]

FYI, basic support is almost ready, it will be added in 0.31 as planned.

For now it will be enabled for any function which is not decorated with @arg. At first it will be limiting but later I'll add the Annotated[x, ExtraParams[...]] feature and you won't need the decorators at all.

Upd.: supporting Annotated means dropping support for Python 3.8, so I'd do it in 0.32 perhaps — was planning to wait until EOY 2024 but it doesn't really make sense, and neither it makes sense to pack such a radical change into 0.31.

[peterjc]

I take it "Goodbye decorators" and "Here’s what Argh is heading for (around 2024)." hasn't quite happened yet?

https://argh.readthedocs.io/en/latest/the_story.html#goodbye-decorators

I was looking at how to set the help test for an option...