Moving off optparse - Githubissues

pradyunsg commented 7 years ago

@pypa/pip-committers Thoughts on switching over to using argparse or even click for option parsing?

xavfernandez commented 7 years ago

Since optparse is deprecated it would make sense to switch to argparse. But I think @dstufft was planning to switch to click.

pradyunsg commented 7 years ago

+1 for click from me.

I'm pretty sure there'd be some edge case the switch would break but that's fine.

thechief389 commented 6 years ago

I think you would be better off with argparse as it's in the standard library and you don't have to add a new dependency to it. It also has the argcomplete library which is good for tab completion.

Also, read this and see what you think: http://xion.io/post/programming/python-dont-use-click.html

pradyunsg commented 6 years ago

Summary of the blog post:

click has an hands-off approach and the author feels that approach isn't appropriate when the CLI is your user interface.

That's a fair argument, made in a fairly emotional tone IMO. (maybe that python-dev thread making me look at things this way)

I'm happy with both options and likely argparse is going to be easier to switch to anyways.

RonnyPfannschmidt commented 6 years ago

it should be noted that argparse has a few pretty nasty "bydesign" bugs around automatically aliasing the prefix pf an argument to the full version, its only opt out in 3.6* and there is no fix for older pythons

thechief389 commented 6 years ago

But nevertheless, argparse is the best choice.

pradyunsg commented 6 years ago

@RonnyPfannschmidt do you have any links? My Googling couldn't find anything... :/

RonnyPfannschmidt commented 6 years ago

https://docs.python.org/3/library/argparse.html#allow-abbrev - sorry for quoting the python version wrong, it was added in 3.5

thechief389 commented 5 years ago

I'm working on moving to click. Is there anyone else doing it?

pradyunsg commented 5 years ago

I've done a bit of work in the past but ran out of time; mostly was just trying to figure out what all is worth investigating, how things could be done and potential improvements before actually starting working on code.

I do have some notes I can share -- none of what's there in the notes is set in stone but might be a good place to start from. You don't need to respond to every hypothetical in it but those should be figured out as we do this. :)

Please feel free to discuss what you're thinking here.

Investigate:

Configuration interaction
- Context Defaults?
- Does the behavior map exactly?
How click's unicode support affects us: https://click.palletsprojects.com/en/7.x/python3/
Utilizing the *_option decorators: Is it worth the effort to bodge-use them when we're not doing decorators?
Argument? Or Context.allow_extra_args? Which is better for us?
How do we use Context in our context? ;)
Backwards compatibility?
- Do we want an "opt-in", "opt-out", "drop" approach here? That's a 3 release cycle removal.
- But look at how much might break via tests to determine the exact flow.
Using click's utilities (in place of our own/through dependencies)
- get_terminal_size
- prompt
- progressbar?
- edit? Maybe?
- echo? I doubt.
- style? Can do but should we?
- get_app_dir? I doubt.
- echo_via_pager --> should help with #5746
click-contrib has some fancy things
- https://github.com/click-contrib/click-completion: Yes.
- https://github.com/click-contrib/click-man: should do
- How to replace existing man page generation? Ping stakeholders on original PR/issue to ask for feedback.
- https://github.com/click-contrib/click-plugins: BIG FEATURE request -- look at this if we do go that route sometime.
- https://github.com/click-contrib/click-configfile: Is this relevant to us?
- https://github.com/click-contrib/click-didyoumean: Replace our custom logic?

Plan:

Rework the existing command setup to have the same "structure".
No decorators.
- Call a method / list of Option in pip.commands? pip.cli.cmdoptions?
Do A/B testing: use an env var to have conditional for how they're parsed
- need to edit tox to pass whatever env var

To be used parts of the API:

Command
Group
Option
Context? (see above)

Avoid direct use:

ASIDE: why aren't these marked more visibly / separated better? File an issue?
CommandCollection
Parameter
BaseCommand
MultiCommand

pradyunsg commented 5 years ago

Hey @thechief389! Have you made any progress on this?

thechief389 commented 5 years ago

I had trouble trying to vendor all the libraries and their dependencies for click and click-completion.

techtonik commented 5 years ago

Another argument against click. After spending 15 minutes I haven't found the code to implement help in this manner:

<command> -h
<command> --help
help <command>
help <command> -h
help <command> --help
help

There 4. 5. 6. are equivalent to help -h or help --help.

davidism commented 5 years ago

Hello, I'm one of the Pallets (Click) core maintainers. I'm happy to answer questions you have, although I'd ask that you ping me here or in our Discord rather than opening up issues for questions in the repo.

I think we're already set up to be vendorable, and if not we can fix that. If you notice any bugs while trying to integrate Click with Pip, please let us know.

davidism commented 5 years ago

The basic implementation of a help name command, although it should be improved:

@cli.command()
@click.argument("name")
@click.pass_context
def help(ctx, name):
    command = cli.get_command(ctx, name)
    click.echo(command.get_help(ctx))

pradyunsg commented 5 years ago

Thanks for pitching in here @davidsm! :)

I'll ping here if and when I work on this. :)

cjerdonek commented 5 years ago

Because of the size of the code base, I think it's important to have an implementation plan that would let us make progress on this incrementally.

pradyunsg commented 5 years ago

We can't really have half our parsing be optparse and the other half be click. I think you know that. ;) I think you mean what the plan is for separating concerns within pip's codebase about this to ease the actual "switch". My current thought process for that is:

modify pip's Commands to stop passing "options" to other internal objects (like to pip_version_check).
- mypy annotations should help here, though I don't think they'll cover 100%.
create custom Option classes and adding code to convert them into concrete optparse variants would make life easier later on.
it should be not-too-difficult to create an optparse.Values-like object using click's higher-level classes to be passed to Commands

PS: it's past 1 am. Don't quote me on these. :)

cjerdonek commented 5 years ago

We can't really have half our parsing be optparse and the other half be click.

That's not what I was saying.. I was saying I think it's important for us to have a way to do it that lets us make changes to the code gradually over time. There can be many possible approaches.

For example, maybe there is a way to add a translation / compatibility layer to have whatever we're migrating to (argparse, click, or whatever) accept optparse options. That would let us switch the options one at a time from optparse to the new format.

The point is to avoid a massive PR or massive amounts of code getting turned on at once. The approach should let us incrementally add new code that is getting used as we add it.

cjerdonek commented 5 years ago

modify pip's Commands to stop passing "options" to other internal objects (like to pip_version_check). ... it should be not-too-difficult to create an optparse.Values-like object using click's higher-level classes to be passed to Commands

Since the options objects are just bags of properties, it seems like it would still work for them to be passed around as long as the interfaces of the objects being passed around remain the same (e.g. same attribute names, etc).

There could be other reasons to stop passing options objects around (e.g. to make dependencies more explicit or to reduce coupling), but it doesn't seem like it's something that has to be done.

RonnyPfannschmidt commented 5 years ago

from our experiences in pytest im relatively confident to claim that a basic command setup for argparse for example is relatively simple to do,

i suspect it would have similar complexity for pip (which didn't exist when re we revamped argument parsing in pytest)

however pip has a major advantage over pytest - the api is not public, so experiments and mistakes cant hit reasonable users at the api level

i also believe it would be a great help to layer things nicely and decouple the internals from details of the command line parser

techtonik commented 5 years ago

i also believe it would be a great help to layer things nicely and decouple the internals from details of the command line parser

In other words, switching from optparse should be just a matter of swapping single cli file.

RonnyPfannschmidt commented 5 years ago

I distance myself from that reprasing of different meaning as for the time being its not clear how things will actually be

Its entirely possible that the ROI of a partial refactoring followed by a change is better than going all the way

pradyunsg commented 5 years ago

The point is to avoid a massive PR or massive amounts of code getting turned on at once. The approach should let us incrementally add new code that is getting used as we add it.

This, I agree with. I pretty weary of such PRs now.

There could be other reasons to stop passing options objects around (e.g. to make dependencies more explicit or to reduce coupling), but it doesn't seem like it's something that has to be done.

Yea, this is unrelated to us switching over from optparse. To digress for a bit, it's still something we should do IMO, to make data dependencies clearer.

Thinking about this in a less sleepy state, I think it should be possible to do optparse -> click conversion at the "Command" level. Then, we can start swapping out cmdoptions and sub-command parsers with their click equivalents.

techtonik commented 5 years ago

@pradyunsg looks like you've already decided to go for click. I still don't see that this decorator approach leads to readable code. It doesn't give a good overview about the whole CLI interface, and couples the whole codebase to click.

pradyunsg commented 5 years ago

I already stated that I don't want to be using decorators: https://github.com/pypa/pip/issues/4659#issuecomment-441323813. Click has other abstractions that I'd prefer to use.

Unless we drop Python 2 support, I don't think we'll be able to use argparse due to the behavior associated with allow_abrev.

thechief389 commented 5 years ago

Pipenv uses click and it works well.

techtonik commented 5 years ago

Poetry looks better and uses cleo.

pradyunsg commented 5 years ago

It's pretty simple to use click's underlying classes to make a CLI like pip's. The more fun part is figuring out how the transition for pip's optparse based code to have click options and making commands get Values-like objects, as @cjerdonek pointed out.

Click to reveal code-dump

```py import inspect import functools import click from .commands import get_commands def call_with_context(func): return func(click.get_current_context()) def create_click_command_from_our_command(our_command): # Convert the options; this bit has to be figured out. params = [ click.Option(opt.param_decls, **opt.kwargs) for opt in our_command.options ] # Construct the click command cmd = click.Command( our_command.name, short_help=inspect.getdoc(our_command), callback=functools.partial(call_with_context, our_command.run), params=params, ) return cmd def main(): cli = click.Group() for our_command in get_commands(): cli.add_command(create_click_command_from_our_command(our_command)) cli.main(prog_name="pip") ```

techtonik commented 5 years ago

Click doesn't support custom formatting - https://github.com/pallets/click/issues/561

davidism commented 5 years ago

There are plenty of ways to override formatting now, adding the ability to override the full formatter class (the issue you linked) is an extension to what's already possible.

thechief389 commented 5 years ago

Why would we need to customize formatting?

Sent from my iPhone

On Feb 9, 2019, at 11:02 PM, anatoly techtonik notifications@github.com wrote:

Click doesn't support custom formatting - pallets/click#561

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

pradyunsg commented 4 years ago

Here's a rough draft of what I'd imagined we'd end up with in terms of how the commands' option declaration looks like. I'm not looking for feedback / inputs on this at this time. I'm noting it here in case someone else comes around to work on this and would like to know a rough target of what I want us to achieve.

...
from pip._internal.cli.base_command import Command
from pip._internal.cli.options import common_options, Choice, IndexOptionsMixin, Option
...

class ListCommand(Command, IndexOptionsMixin):
    """
    List installed packages, including editables.

    Packages are listed in a case-insensitive sorted order.
    """

    usage = """
      %prog [options]"""

    options = [
        Option(
            "-o", "--outdated",
            is_flag=True,
            help="List outdated packages",
        ),
        Option(
            "-u", "--uptodate",
            is_flag=True,
            help="List up-to-date packages",
        ),
        Option(
            "-e", "--editable",
            is_flag=True,
            help="List editable projects.",
        ),
        Option(
            "-l",
            "--local",
            is_flag=True,
            help=(
                "If in a virtualenv that has global access, do not list "
                "globally-installed packages."
            ),
        ),
        Option(
            "--user",
            is_flag=True,
            help="Only output packages installed in user-site."
        ),
        common_options.list_path,
        Option(
            "--pre",
            is_flag=True,
            help=(
                "Include pre-release and development versions. By default, "
                "pip only finds stable versions."
            ),
        ),
        Option(
            "--format",
            type=Choice(["columns", "freeze", "json"]),
            default="columns",
            dest="list_format",
            help=(
                "Select the output format among: columns (default), freeze, or json"
            ),
        ),
        Option(
            "--not-required",
            is_flag=True,
            help="List packages that are not dependencies of installed packages.",
        ),
        Option(
            "--include-editable/--exclude-editable",
            is_flag=True,
            help="Include editable package from output.",
        ),
    ]

    # No need for wrangling with `self.cmd_opts` in `__init__`
    # Rest of the command definition follows immediately.

sinscary commented 4 years ago

@pradyunsg Is this still up for development?

pradyunsg commented 4 years ago

@sinscary certainly!

sinscary commented 4 years ago

@pradyunsg In that case I would like to give it a try.

pradyunsg commented 4 years ago

Please do! In case you want to have a voice / video chat about this, free free to reach out to me over email. :)

janluke commented 3 years ago

Shameless plug: in case you need additional features for Click, like argparse argument groups, you may consider Cloup. To my knowledge, it's the only package that allows to define option groups without using decorators with Click.

pradyunsg commented 2 years ago

Well, here's a fun thing that's going to make this a bit more painful:

I ran pip wheel setuptools --build /tmp/fooooo (we removed the --build/--build-dir options in 21.3) and optparse has been a very helpful parser and completed that to --build-option. From a breakpoint at the start of WheelCommand.run:

(Pdb) options.build_options
['/tmp/fooooo']
(Pdb) sys.argv
['/Users/pgedam/Developer/totally-normal-nothing-to-see-here-project/.venv/bin/pip', 'wheel', 'setuptools', '--build', '/tmp/fooooo']

🤦🏽

Shivansh-007 commented 2 years ago

How about using sdispater/cleo @pradyunsg? It supports the class-based approach that we are wanting to have, along with some other pretty features like text colouring.

How poetry uses it for its CLI - https://github.com/python-poetry/poetry/blob/6485bc23d6497c7731e0f1a635f960b33f2ae99e/src/poetry/console/commands/export.py#L9-L38

uranusjr commented 2 years ago

Actually from the above discussion argparse is by far the best choice. The only blocker against it has been removed now pip has dropped all Python <3.6. And since it’s stdlib, nothing can really beat it at this point (plus migrating from optparse to argparse might actually be the least work due to their similarities in interface design).

Shivansh-007 commented 2 years ago

For argparse, we would need to build some features from scratch as we have currently done for optparse, IMO command-line libraries like click and cleo are doing a much great job at the interface they are providing for building apps and have pretty good features.

Migrating to cleo won't be a big pain as they share a bit of similarity, the way we currently have them written is similar to cleo's style.

uranusjr commented 2 years ago

I wouldn’t mind reviewing a POC if you feel it is easy enough to do. Note that we won’t be using much of the features apart from command line parsing (e.g. colors) in either case since those are already done with other mechanisms.

notatallshaw commented 1 year ago

FWIW it seems that while optparse is "deprecated" it's clearly the opinion, in this discussion about an argparse bug, that optparse will never be removed and argparse development itself has largely slowed down in the face of design issues,

slightlybelowzen commented 9 months ago

I'd be happy to work on a small POC for this (using argparse) if there's still interest in moving off of optparse. We can decide where to go from there.

pradyunsg commented 9 months ago

There is, and please feel welcome to do a PoC!

slightlybelowzen commented 9 months ago

Also I don't know if I'm missing something but in the docs it says, the cli/ subpackage is implemented using argparse which seems off? https://github.com/pypa/pip/blob/51de88ca6459fdd5213f86a54b021a80884572f9/docs/html/development/architecture/anatomy.rst?plain=1#L75

Hopefully after this PR is merged, we won't need to update it😁!

uranusjr commented 8 months ago

The doc entry is mostly likely a typo, it’s added much later than the actual implementation.

pypa / pip

Moving off optparse #4659