search

aquaproj / aqua

Declarative CLI Version manager written in Go. Support Lazy Install, Registry, and continuous update with Renovate. CLI version is switched seamlessly
https://aquaproj.github.io
885 stars 39 forks source link

Support installing packages by `pip install` #2128

Open suzuki-shunsuke opened 1 year ago

suzuki-shunsuke commented 1 year ago

Feature Overview

Support installing packages by pip install.

Why is the feature needed?

To support packages written in Python.

Workaround

Install tools with other tools.

  1. pyenv and virtualenv
  2. homebrew

Example Code

registry.yaml

packages:
  - name: pypi.org/pre-commit
    type: pip
    pip_name: pre-commit
    files:
      - name: pre-commit

Reference

suzuki-shunsuke commented 1 year ago

pip install supports --root option.

  --user                      Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation
                              for site.USER_BASE for full details.)
  --root <dir>                Install everything relative to this alternate root directory.
```console $ pip --version pip 21.1.2 from /Users/shunsuke-suzuki/.pyenv/versions/3.9.5/lib/python3.9/site-packages/pip (python 3.9) $ pip install --help Usage: pip install [options] [package-index-options] ... pip install [options] -r [package-index-options] ... pip install [options] [-e] ... pip install [options] [-e] ... pip install [options] ... Description: Install packages from: - PyPI (and other indexes) using requirement specifiers. - VCS project urls. - Local project directories. - Local or remote source archives. pip also supports installing from "requirements files", which provide an easy way to specify a whole environment to be installed. Install Options: -r, --requirement Install from the given requirements file. This option can be used multiple times. -c, --constraint Constrain versions using the given constraints file. This option can be used multiple times. --no-deps Don't install package dependencies. --pre Include pre-release and development versions. By default, pip only finds stable versions. -e, --editable Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url. -t, --target Install packages into . By default this will not replace existing files/folders in . Use --upgrade to replace existing packages in with new versions. --platform Only use wheels compatible with . Defaults to the platform of the running system. Use this option multiple times to specify multiple platforms supported by the target interpreter. --python-version The Python interpreter version to use for wheel and "Requires-Python" compatibility checks. Defaults to a version derived from the running interpreter. The version can be specified using up to three dot-separated integers (e.g. "3" for 3.0.0, "3.7" for 3.7.0, or "3.7.3"). A major- minor version can also be given as a string without dots (e.g. "37" for 3.7.0). --implementation Only use wheels compatible with Python implementation , e.g. 'pp', 'jy', 'cp', or 'ip'. If not specified, then the current interpreter implementation is used. Use 'py' to force implementation-agnostic wheels. --abi Only use wheels compatible with Python abi , e.g. 'pypy_41'. If not specified, then the current interpreter abi tag is used. Use this option multiple times to specify multiple abis supported by the target interpreter. Generally you will need to specify --implementation, --platform, and --python-version when using this option. --user Install to the Python user install directory for your platform. Typically ~/.local/, or %APPDATA%\Python on Windows. (See the Python documentation for site.USER_BASE for full details.) --root Install everything relative to this alternate root directory. --prefix Installation prefix where lib, bin and other top-level folders are placed --src Directory to check out editable projects into. The default in a virtualenv is "/src". The default for global installs is "/src". -U, --upgrade Upgrade all specified packages to the newest available version. The handling of dependencies depends on the upgrade-strategy used. --upgrade-strategy Determines how dependency upgrading should be handled [default: only-if-needed]. "eager" - dependencies are upgraded regardless of whether the currently installed version satisfies the requirements of the upgraded package(s). "only-if-needed" - are upgraded only when they do not satisfy the requirements of the upgraded package(s). --force-reinstall Reinstall all packages even if they are already up-to-date. -I, --ignore-installed Ignore the installed packages, overwriting them. This can break your system if the existing package is of a different version or was installed with a different package manager! --ignore-requires-python Ignore the Requires-Python information. --no-build-isolation Disable isolation when building a modern source distribution. Build dependencies specified by PEP 518 must be already installed if this option is used. --use-pep517 Use PEP 517 for building source distributions (use --no-use-pep517 to force legacy behaviour). --install-option Extra arguments to be supplied to the setup.py install command (use like --install-option="--install-scripts=/usr/local/bin"). Use multiple --install-option options to pass multiple options to setup.py install. If you are using an option with a directory path, be sure to use absolute path. --global-option Extra global options to be supplied to the setup.py call before the install or bdist_wheel command. --compile Compile Python source files to bytecode --no-compile Do not compile Python source files to bytecode --no-warn-script-location Do not warn when installing scripts outside PATH --no-warn-conflicts Do not warn about broken dependencies --no-binary Do not use binary packages. Can be supplied multiple times, and each time adds to the existing value. Accepts either ":all:" to disable all binary packages, ":none:" to empty the set (notice the colons), or one or more package names with commas between them (no colons). Note that some packages are tricky to compile and may fail to install when this option is used on them. --only-binary Do not use source packages. Can be supplied multiple times, and each time adds to the existing value. Accepts either ":all:" to disable all source packages, ":none:" to empty the set, or one or more package names with commas between them. Packages without binary distributions will fail to install when this option is used on them. --prefer-binary Prefer older binary packages over newer source packages. --require-hashes Require a hash to check each requirement against, for repeatable installs. This option is implied when any package in a requirements file has a --hash option. --progress-bar Specify type of progress to be displayed [off|on|ascii|pretty|emoji] (default: on) --no-clean Don't clean up build directories. Package Index Options: -i, --index-url Base URL of the Python Package Index (default https://pypi.org/simple). This should point to a repository compliant with PEP 503 (the simple repository API) or a local directory laid out in the same format. --extra-index-url Extra URLs of package indexes to use in addition to --index-url. Should follow the same rules as --index-url. --no-index Ignore package index (only looking at --find-links URLs instead). -f, --find-links If a URL or path to an html file, then parse for links to archives such as sdist (.tar.gz) or wheel (.whl) files. If a local path or file:// URL that's a directory, then look for archives in the directory listing. Links to VCS project URLs are not supported. General Options: -h, --help Show help. --isolated Run pip in an isolated mode, ignoring environment variables and user configuration. -v, --verbose Give more output. Option is additive, and can be used up to 3 times. -V, --version Show version and exit. -q, --quiet Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels). --log Path to a verbose appending log. --no-input Disable prompting for input. --proxy Specify a proxy in the form [user:passwd@]proxy.server:port. --retries Maximum number of retries each connection should attempt (default 5 times). --timeout Set the socket timeout (default 15 seconds). --exists-action Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort. --trusted-host Mark this host or host:port pair as trusted, even though it does not have valid or any HTTPS. --cert Path to PEM-encoded CA certificate bundle. If provided, overrides the default. See 'SSL Certificate Verification' in pip documentation for more information. --client-cert Path to SSL client certificate, a single file containing the private key and the certificate in PEM format. --cache-dir Store the cache data in . --no-cache-dir Disable the cache. --disable-pip-version-check Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index. --no-color Suppress colored output. --no-python-version-warning Silence deprecation warnings for upcoming unsupported Pythons. --use-feature Enable new functionality, that may be backward incompatible. --use-deprecated Enable deprecated functionality, that will be removed in the future. ```

e.g.

 $ pip install --root pip-root pre-commit
$ ls pip-root/Users/shunsuke-suzuki/.local/bin 
identify-cli  nodeenv  pre-commit  virtualenv
suzuki-shunsuke commented 1 year ago

Directory structure

<AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<package name>/<version>/bin/<command>
AQUA_ROOT_DIR/pkgs/
  pip/
    pypi.org/
      pre-commit/
        3.3.3/
          bin/
            pre-commit
          lib/
suzuki-shunsuke commented 1 year ago

Renovate

Pypi datasource

https://docs.renovatebot.com/modules/datasource/pypi/

suzuki-shunsuke commented 1 year ago

Package name

Packages in https://pypi.org/ should start with pypi.org/ so that aqua-renovate-config can get versions from pypi datasource and aqua g gets the version from https://pypi.org/ if API exists.

In the repository aquaproj/aqua-registry, pip packages should be located in pkgs/pypi.org similar to crates.io.

suzuki-shunsuke commented 1 year ago

pip install supports --root option.

--target is more useful than --root.

$ pip install -t target pre-commit
$ ls target/bin 
identify-cli  nodeenv  pre-commit  virtualenv
suzuki-shunsuke commented 1 year ago
suzuki-shunsuke commented 1 year ago

⚠️ It failed to execute pre-commit installed by pip install --target command

Hmm. I installed pre-commit by pip install --target target pre-commit, but it failed to execute pre-commit command.

$ python --version
Python 3.10.5

$ pip --version
pip 22.2 from /Users/shunsukesuzuki/.pyenv/versions/3.10.5/lib/python3.10/site-packages/pip (python 3.10)
$ pip install --target target pre-commit
$ ./target/bin/pre-commit --help
Traceback (most recent call last):
  File "/Users/shunsukesuzuki/Documents/test/pip/./target/bin/pre-commit", line 5, in <module>
    from pre_commit.main import main
ModuleNotFoundError: No module named 'pre_commit'

target/bin/pre-commit

#!/Users/shunsukesuzuki/.pyenv/versions/3.10.5/bin/python3.10
# -*- coding: utf-8 -*-
import re
import sys
from pre_commit.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

pre_commit exists in target.

The environment variable PYTHONPATH is needed?

Succeeded.

$ env PYTHONPATH=$PWD/target ./target/bin/pre-commit --help
usage: pre-commit [-h] [-V]
                  {autoupdate,clean,gc,init-templatedir,install,install-hooks,migrate-config,run,sample-config,try-repo,uninstall,validate-config,validate-manifest,help,hook-impl}
                  ...

positional arguments:
  {autoupdate,clean,gc,init-templatedir,install,install-hooks,migrate-config,run,sample-config,try-repo,uninstall,validate-config,validate-manifest,help,hook-impl}
    autoupdate          Auto-update pre-commit config to the latest repos' versions.
    clean               Clean out pre-commit files.
    gc                  Clean unused cached repos.
    init-templatedir    Install hook script in a directory intended for use with `git config init.templateDir`.
    install             Install the pre-commit script.
    install-hooks       Install hook environments for all environments in the config file. You may find `pre-commit install
                        --install-hooks` more useful.
    migrate-config      Migrate list configuration to new map configuration.
    run                 Run hooks.
    sample-config       Produce a sample .pre-commit-config.yaml file
    try-repo            Try the hooks in a repository, useful for developing new hooks.
    uninstall           Uninstall the pre-commit script.
    validate-config     Validate .pre-commit-config.yaml files
    validate-manifest   Validate .pre-commit-hooks.yaml files
    help                Show help for a specific command.

options:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit

I'm not sure if this is a correct solution.

suzuki-shunsuke commented 1 year ago

⚠️ the behaviour of pip command depends on Python and pip version

BTW, the behaviour of pip command depends on Python and pip version. This means pip package strongly depends on the environment and the reproductivity is low.

We have no plan (probably we won't) to support managing isolated Python environment like pyenv and virtualenv.

If the project uses tools such as pyenv and virtualenv, I guess it's better to manage pip packages without aqua.

suzuki-shunsuke commented 1 year ago

Get the list of available versions

https://stackoverflow.com/a/27239645/6364492

curl "https://pypi.org/pypi/pre-commit/json" | jq -r '.releases | keys[]'
suzuki-shunsuke commented 1 year ago

Document (Draft)

Support installing packages by pip install

2128 #2142 aqua >= v2.11.0

Feature Overview

Support installing packages by pip install.

Why is the feature needed?

To support packages written in Python.

Requirements

aqua doesn't install these requirements automatically. Please install them yourself.

This feature depends on versions of Python and pip. We verified this feature with the following versions.

$ python --version
$ pip --version

Getting Started

Let's install pre-commit by aqua.

aqua init
aqua g -i pypi.org/pre-commit

You can also select the version by -s option.

aqua g -i -s pypi.org/pre-commit
aqua i -l
aqua which pre-commit
pre-commit --version

aqua-renovate-config

https://github.com/aquaproj/aqua-renovate-config

From 1.8.0, aqua-renovate-config supports updating pypi packages.

⚠️ Note that package names must be pypi.org/<pypi package name>.

e.g. pypi.org/pre-commit.

How to add new pypi packages to Standard Registry

Please send a pull request to https://github.com/aquaproj/aqua-registry . Package names must be pypi.org/<pypi package name>.

e.g. pypi.org/pre-commit.

How does it work?

⚠️ This includes details of the internal implementation, which may be changed without notification. Please skip this section if you're not interested in the detail. You can use pypi packages even if you don't know this.

pypi packages are installed in <AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version>, and executable files are installed in <AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version>/bin/<command>.

aqua internally runs python -m pip install commands.

python -m pip install --target  "<AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version>" "<pypi package>==<version>"

aqua g -s gets the list of pypi package versions from the endpoint https://pypi.org/pypi/<pypi package name>/json.

<AQUA_ROOT_DIR>/pkgs/pip/pypi.org/<pypi package name>/<version> is added to the environment variable PYTHONPATH when pypi packages are executed.

Registry

e.g.

packages:
  - type: pypi
    pypi_name: pre-commit

type must be pypi. pypi_name is required. Other fields are optional. The above setting is equivalent to the following setting.

packages:
  - name: pypi.org/pre-commit
    type: pypi
    pypi_name: pre-commit
    files:
      - name: pre-commit

pypi packages don't support the following fields.

suzuki-shunsuke commented 1 year ago

TODO

suzuki-shunsuke commented 1 year ago

I decided to rename the package type name pip to pypi.

suzuki-shunsuke commented 1 year ago

Hi @Gowiem ,

I published a pre-release version v2.11.0-3 v2.11.0-4. Could you try this? If you have any feedback, please let me know.

Document: https://github.com/aquaproj/aqua/issues/2128#issuecomment-1657001257

aqua update-aqua v2.11.0-4
aqua -v
mkdir workspace
cd workspace
aqua init
vi aqua.yaml

aqua.yaml

---
registries:
- type: standard
  ref: cd6610dcdee69814d8f72e06e9557d42abad4298 # https://github.com/aquaproj/aqua-registry/pull/14185
packages:

Install pre-commit.

aqua g -i pypi.org/pre-commit
aqua i -l

Please check if pre-commit installed by aqua is executed.

command -v pre-commit

If pre-commit installed by other than aqua is executed, you have some options.

  1. Uninstall it once
  2. Fix the order of the environment variable PATH
  3. Run pre-commit via aqua exec command. aqua exec -- pre-commit

Then please confirm that pre-commit is available.

pre-commit --help
suzuki-shunsuke commented 1 year ago

📝 What is the effect of using python -m pip instead of just pip?

suzuki-shunsuke commented 1 year ago

⚠️ Some pypi packages don't work well

I'm trying some tools that can be installed by pip install command.

https://github.com/topics/cli?l=python

Unfortunately, some tools don't work well.

textual

https://github.com/Textualize/textual

ModuleNotFoundError: No module named 'aiohttp'
```console $ textual console Traceback (most recent call last): File "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/pypi/pypi.org/textual/0.9.1/bin/textual", line 8, in sys.exit(run()) File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/shunsukesuzuki/.local/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/pypi/pypi.org/textual/0.9.1/textual/cli/cli.py", line 23, in console from textual.devtools.server import _run_devtools File "/Users/shunsukesuzuki/.local/share/aquaproj-aqua/pkgs/pypi/pypi.org/textual/0.9.1/textual/devtools/server.py", line 4, in from aiohttp.web import run_app ModuleNotFoundError: No module named 'aiohttp' ```
suzuki-shunsuke commented 1 year ago

I'm adding some pypi packages.

suzuki-shunsuke commented 1 year ago

⚠️ Should we really introduce this feature?

I implemented this feature, but now I hesitate to introduce it because other tools such as pyenv, virtualenv, and pipx seem to be better than aqua. I'm not familiar with Python, but Python has already several version management tools. They support managing Python and dependencies in isolated environments, and Renovate supports updating them.

I tried to install some pypi packages by aqua, but some of them don't work well because they depend on other libraries. https://github.com/aquaproj/aqua/issues/2128#issuecomment-1657098894 I don't want to handle these troubles.

aqua has various advantages compared with other tools such as Homebrew and asdf, but in case of pypi packages, aqua doesn't have so many advantages compared with other tools such as pyenv, virtualenv, and pipx.

pypi packages can be installed by pip easily, but to make them available by aqua, we need to add them to Registries. It's a little bothersome.

dudicoco commented 1 year ago

@suzuki-shunsuke note that pipx is not suitable for project specific tool versions, see https://github.com/pypa/pipx/issues/1031

Perhaps aqua can provide a wrapper for both pipx and npx to install pip and node packages declaratively?

Aqua could run pipx under the hood and then just add the created binary to its path, for example /Users/xxxx/.local/pipx/venvs/copier/bin/copier

dudicoco commented 1 year ago

So rethinking this, i'm not sure pipx is a good solution since I believe it can produce non-deterministic builds: https://github.com/pypa/pipx/issues/1050

I believe that in order to make aqua work with pip packages and make them reproducible, aqua would need to do the following:

  1. Use pyenv to set a consistent python version
  2. Use an automatically generated lock file for each version of a pip package - this can be part of the workflow which updates the registry. Options for generation the lock file:
  3. Install the pip package using virtualenv within the aqua cache directory
  4. Add the executable to the path
suzuki-shunsuke commented 1 year ago

I see. Thank you. I misunderstood pipx.

How about direnv?

https://github.com/direnv/direnv/wiki/Python

.envrc

layout python

requirements.txt

pre-commit==3.3.3
direnv allow
pip install -r requirements.txt
$ command -V pre-commit
pre-commit is /Users/shunsukesuzuki/Documents/test/foo/.direnv/python-3.10/bin/pre-commit

$ pre-commit -V
pre-commit 3.3.3

To specify Python version, direnv + pyenv is useful.

https://github.com/direnv/direnv/wiki/Python#pyenv

dudicoco commented 1 year ago

Yeah i've also considered direnv as an alternative, there are a few issues though:

  1. The user must manage the requirements.txt file, also if you want a few packages in different versions you might have to manage multiple requirements.txt files due to possible conflicting versions.
  2. I'm not sure if direnv sets a shared cache so if i've installed package x with version 1.0.0 in repo A I might not be able to use the cached package in repo B as well. Even if this is possible, the repos might have different dependencies in the requirements.txt file which will make them incompatible.
  3. Just like in aqua, we do not want the user to run pip install manually, we would like it to run automatically and i'm not sure if direnv can do that