Identify projects which could use this library

[di]

Goals

1. Create a list of projects that currently use pip's internal API
2. Identify which use cases are not covered by pip-api
3. Open issues for each use case
4. Implement API for each use case
5. Open PR to migrate target project to pip-api

Projects successfully migrated to pip-api:

• https://github.com/peterbe/hashin/
• https://github.com/di/pytest-reqs/

Projects currently using pip's internal API

(Some likely candidates here: https://www.wheelodex.org/projects/pip/rdepends/)

• https://github.com/jazzband/pip-tools/
  • This is a big one: https://github.com/jazzband/pip-tools/blob/master/piptools/_compat/pip_compat.py

[phodge]

I would love to have a stable Python API for my dotfiles management project. (See https://github.com/phodge/homely/blob/master/homely/pipinstall.py)

My wishlist of features:

• get_installed_version(package_name, user=False): returns the version number of the currently installed version of a specific package, or None if it isn't installed. If user=True then it only checks the user-specific package list in the current user's home directly (i.e., packages installed by pip install --user). If user=False then it only checks the packages installed at system level.
• get_latest_version(package_name): returns the version number of the latest version of the package as available on pypi.org. The package doesn't have to be installed for this to work.
• get_dependencies(package_name): Returns the list of [package_name, version_specifier] for packages that are direct prerequisites for the named package.
• install_package(package_name, user=False, install_dependencies=True): Install a package, i.e. pip install <package_name>, or pip install --user <package_name> when user=True. If install_dependencies=False and some of the dependencies are missing, then the installation should fail and raise an exception.

[di]

Glad to hear you're considering using this project! Happy to help.

get_installed_version(package_name, user=False)

This could be done with the pip_api.installed_distributions() function. This doesn't currently support scoping to user packages, but that would be a great addition.

get_latest_version(package_name)

I don't see why pip would need to get involved here. This can be done with a simple request to PyPI's JSON API.

get_dependencies(package_name)

Are you doing this now with pip? It's nearly impossible to get 100% correct, see "Why PyPI Doesn't Know Your Projects Dependencies".

install_package(package_name, user=False, install_dependencies=True)

I think this makes sense and can be added, probably as pip_api.install(...)!

[belm0]

Re. https://github.com/timothycrosley/isort/issues/924, I'd like to see the isort package use pip-api.

[di]

@belm0 It doesn't seem like this project would require any new APIs, unless I'm mistaken. I've left a comment here: https://github.com/timothycrosley/isort/issues/924#issuecomment-484280840

[jre21]

@di I've been experimenting with using isort's pip-api-based requirements-file parsing, and I've been running into a snag. If parse_requirements() fails to parse any line in a file, it fails the entire file, whereas for the isort use-case, it would be much more useful to parse as many lines as possible and discard the rest, in the vein of #9 . Would you be amenable to either reviving that pull request, or to me sending pr's for individual corner cases that I would like handled more gracefully?

[di]

Hi @jre21, thanks for the comment. I think you're probably right that pip-api should allow for the original behavior here. The changes in #9 are probably acceptable, but I haven't had a chance to review. Could you take a look and see if it solves your issue?

[jre21]

I think I've found a cleaner way to parse the particular cases I was having trouble with. Pull request incoming.

[jayvdb]

9 is rebased again.

[jayvdb]

Worth also considering, when should this library not be used? and what should be used instead?

Given the conflicting parsers and structs in setuptools, pip, and packaging, (especially older releases), anyone wanting a stable requirements parser which aims to parse the oddities of all of those, and new formats like pipfile/pyproject.toml, they should really be trying to use https://github.com/sarugaku/requirementslib which is part of the new pipenv collection of libraries. It is more permissive, and possibly has some very slight differences to pip internals, but is surely a safer tool than hoping that the installed version of setuptools, pip and packaging are all aligned perfectly.

That then puts pip-api into the more niche area of users who need to access pip's functionality other than requirements parsing, with all of its bells and any of its faults.

Personally I think pip-api should strongly recommend requirementslib for the requirements parsing and structs, even integrating this, and then have a conversion layer from requirementslib structs to the structs of the installed pip version, so that the client can be happily working with requirementslib and then use its requirements classes with pip-api to invoke pip functionality that were expecting classes from packaging or from deep inside pip. That may eventually lead to pipenv being able to use pip-api as a cleaner looser coupling to the pip it currently vendors called 'notpip', maybe even leading to the possibility that pip could be de-vendored from pipenv and support versions of pip where pip-api has a satisfactory bridge.

[malikoth]

get_latest_version(package_name)

I don't see why pip would need to get involved here. This can be done with a simple request to PyPI's JSON API.

It certainly could be done, but I would expect that anything pip can do, an abstraction layer that sits on top of it can also do. I am of course referring to running pip list --outdated which makes (presumably) that same query to the JSON API, and I assume then filters the output to actually just print the ones where installed version and latest version don't match. As the pip_api library clearly doesn't want or need to deal with actually printing anything, I certainly wouldn't expect it to do that filtering for me. I definitely wouldn't want it actually making HTTP requests every time I call pip_api.installed_distributions, just as pip list doesn't make that request if you don't pass in the --outdated flag. But having a function that will do it for you seems well in line with the nature of this project to me. Just my $0.02. :)

[di]

I think a function which more or less wraps pip list would definitely be valuable, but I want to be careful to scope the purpose of this package to providing a drop-in replacement for existing uses of pip's internal API. To my knowledge, there is no equivalent to get_latest_version, the closest would probably be iter_packages_latest_infos, which I haven't seen much usage of.

I think get_latest_version would probably belong in the pypi-api project instead which is a WIP. 🙂

[wimglenn]

Hey Dustin, with a few more features this pip-api project could really be useful for johnnydep . I currently call pip in a subprocess, as suggested in Using pip from your program.

The stuff I need:

• List versions available for a given distribution
• Download urls for the releases, preferably with checksums (PyPI seems to offer this in the url fragment).
• Ability to use custom index, if set in pip.conf or env var

cheers

[di]

Hey @wimglenn, thanks for commenting. Would be great to use this project in johnnydep, and in case it's not clear, under the hood pip-api is just using subprocess as described there.

I think similar to the request for get_latest_version above, listing the versions available and getting the download URLs are probably best served by just hitting the PyPI API directly, although I'd be curious to see how you're currently using pip for this.

The configuration stuff is totally valid, I created https://github.com/di/pip-api/issues/44 to capture this.

[wimglenn]

The reason I don't use PyPI API directly there is because I want johnnydep to also see user's pip configuration (e.g. at $EMPLOYER, where your gloablly configured index_url might be a devpi-server with some internal packages). Yeah, I suppose I could also read the configs and stuff, but it's much easier just to let pip do it however it wants to do it.

[di]

Got it, that totally makes sense. What pip commands are you currently using for those two things?

[wimglenn]

For listing versions avail it gets a bogus version and handles the error output. For download urls, scrape debug log events 😞 Lame I know, but pip doesn't really offer you much else here..

[irvinlim]

Hey @di,

I love the idea of this project, but the current list of features is too limited for me to use right now.

My main use case is to manage virtualenvs (perhaps see https://github.com/irvinlim/dotfiles/tree/master/installer), which uses sjkingo/virtualenv-api to programmatically manage virtualenvs, which executes pip in a subprocess using the relevant virtualenv's Python interpreter under the hood, and manually parsing the standard output, basically like what pip_api does.

Currently, I use sjkingo/virtualenv-api to install distributions (pip install), list packages (pip list), and check if a package is already installed. It currently doesn't support pip show.

I was hoping that pip_api could support the above mentioned commands (specifically install and show). Are you planning to work on those commands? If not, what should I look out for when implementing them in order to maximize compatibility with all pip versions?

Additionally, in order to extend the same functionality that sjkingo/virtualenv-api provides, could there be a way to perhaps initialize pip_api with a custom Python path and/or pip path?

[di]

Hi @irvinlim, thanks for the comment! I think the functionality provided by install and show are probably fair game for this library. I think list is probably covered by pip_api.installed_distributions.

One thing to note is that this library is focused on providing a close to drop-in replacement for the internal functions that people might be currently be importing from pip._internal, not necessarily to wrap the CLI 1:1. So it would be ideal to identify said functions rather than the CLI sub-commands.

For show, the goal is to get the metadata about the installed packages. It looks like users might be using search_packages_info() for this. It also seems like using importlib.metadata probably makes more sense here, so we may not want to wrap this at all.

For install, it's a bit of a behemoth and might be a challenge to get right. E.g., what would you expect pip_api.install to return?

[penn5]

For install, it's a bit of a behemoth and might be a challenge to get right. E.g., what would you expect pip_api.install to return?

It could return {"installed_package": "version_code", "package_2": "vc_2"}

[di]

It could return {"installed_package": "version_code", "package_2": "vc_2"}

It doesn't seem like that's what InstallCommand.run() would return, it looks like it returns an integer

I tried to use this the "bad" way, and it's quite complex. I'm not convinced that anyone is actually doing this:

from pip._internal.commands.install import InstallCommand
from pip._internal.models.format_control import FormatControl
from pip._internal.utils.temp_dir import global_tempdir_manager, tempdir_registry

from optparse import Values

options = Values(
    {
        "help": None,
        "isolated_mode": False,
        "require_venv": False,
        "verbose": 0,
        "version": None,
        "quiet": 0,
        "log": None,
        "no_input": False,
        "proxy": "",
        "retries": 5,
        "timeout": 15,
        "exists_action": [],
        "trusted_hosts": [],
        "cert": None,
        "client_cert": None,
        "cache_dir": "/Users/dustiningram/Library/Caches/pip",
        "disable_pip_version_check": False,
        "no_color": False,
        "no_python_version_warning": False,
        "unstable_features": [],
        "features_enabled": [],
        "deprecated_features_enabled": [],
        "requirements": [],
        "constraints": [],
        "ignore_dependencies": False,
        "pre": False,
        "editables": [],
        "target_dir": None,
        "platform": None,
        "python_version": None,
        "implementation": None,
        "abi": None,
        "use_user_site": None,
        "root_path": None,
        "prefix_path": None,
        "build_dir": None,
        "src_dir": "/private/tmp/env/src",
        "upgrade": None,
        "upgrade_strategy": "only-if-needed",
        "force_reinstall": None,
        "ignore_installed": None,
        "ignore_requires_python": None,
        "build_isolation": True,
        "use_pep517": None,
        "install_options": None,
        "global_options": None,
        "compile": True,
        "warn_script_location": True,
        "warn_about_conflicts": True,
        "format_control": FormatControl(set(), set()),
        "prefer_binary": False,
        "require_hashes": False,
        "progress_bar": "on",
        "index_url": "https://pypi.org/simple",
        "extra_index_urls": [],
        "no_index": False,
        "find_links": [],
        "no_clean": False,
    }
)

with global_tempdir_manager():
    c = InstallCommand('name', 'summary')
    c._in_main_context = True
    reg = c.enter_context(tempdir_registry())
    c.tempdir_registry = reg
    c.run(options, ["sampleproject"])

[di]

For folks asking about install in this thread, you may like to know that I've added experimental PEP 650 support in #65, which would eventually allow you to do the following if/when the PEP is finalized:

• pip_api.invoke_install
• pip_api.invoke_uninstall