search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.83k stars 1.87k forks source link

VCS install using https credentials in env variables fails for Pipfile #6195

Open dennisvang opened 2 months ago

dennisvang commented 2 months ago

Issue description

Trying to install a package over https, from a private AWS CodeCommit repo, on an Ubuntu system.

Instead of using git credential.helper, we would like to inject credentials into our Pipfile using env variables.

So, in a bash terminal, we define USERNAME and PASSWORD (url-encoded).

Installing into an empty dir, directly from the command line, using these credentials, appears to work without issue:

pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

This implies that the credentials and user permissions are correct.

However, it turns out the username and password end up in both Pipfile and Pipfile.lock...

That does not sound like a good idea.

Luckily, the docs for Injecting credentials into Pipfile via environment variables say:

Keep in mind that environment variables are expanded in runtime, leaving the entries in Pipfile or Pipfile.lock untouched. This is to avoid the accidental leakage of credentials in the source code.

However, if we try to install from a Pipfile into an otherwise empty dir, using the exact same url with same USERNAME and PASSWORD, we get a status 403 (Forbidden):

# ...
[packages]
mypackage = {editable = true, ref = "main", git = "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage"}
# ...

yields

...
INFO:pip.subprocessor:fatal: unable to access 'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': 
The requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
exited with 128
...

Expected result

I expect installation from Pipfile to work without issue, just like installation from the command line.

Actual result

command line output ```none /tmp/pipenvtest $ pipenv install --verbose Pipfile.lock not found, creating... Locking [packages] dependencies... Building requirements... Resolving dependencies... INFO:pipenv.patched.pip._internal.operations.prepare:Collecting mypackage@ git+https://myusername:****@git-codecommit.eu-west-3.a mazonaws.com/v1/repos/mypackage@main (from -r /tmp/pipenv-2_t8zu99-requirements/pipenv-3cdgwj18-constraints.txt (line 2)) INFO:pipenv.patched.pip._internal.vcs.git:Cloning https://myusername:****@git-codecommit.eu-west-3.amazo naws.com/v1/repos/mypackage (to revision main) to /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amaz onaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 INFO:pip.subprocessor:fatal: unable to access 'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': The requested URL returned error: 403 ERROR:pip.subprocessor:git clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amaz onaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 128 Traceback (most recent call last): File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 455, in resolve results = resolver.resolve(constraints, check_supported_wheels=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/resolver.py", line 76, in resolve collected = self.factory.collect_root_requirements(root_reqs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/factory.py", line 534, in collect_root_requirements reqs = list( ^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req cand = self._make_base_candidate_from_link( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link self._link_candidate_cache = LinkCandidate( ^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/candidates.py", line 294, in __init__ super().__init__( File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/candidates.py", line 157, in __init__ self.dist = self._prepare() ^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/candidates.py", line 223, in _prepare dist = self._prepare_distribution() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso lution/resolvelib/candidates.py", line 305, in _prepare_distribution return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper ations/prepare.py", line 525, in prepare_linked_requirement return self._prepare_linked_requirement(req, parallel_builds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper ations/prepare.py", line 596, in _prepare_linked_requirement local_file = unpack_url( ^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper ations/prepare.py", line 157, in unpack_url unpack_vcs_link(link, location, verbosity=verbosity) File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper ations/prepare.py", line 80, in unpack_vcs_link vcs_backend.unpack(location, url=hide_url(link.url), verbosity=verbosity) File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/ versioncontrol.py", line 608, in unpack self.obtain(location, url=url, verbosity=verbosity) File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/ versioncontrol.py", line 521, in obtain self.fetch_new(dest, url, rev_options, verbosity=verbosity) File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/ git.py", line 276, in fetch_new self.run_command( File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/ versioncontrol.py", line 650, in run_command return call_subprocess( ^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/util s/subprocess.py", line 224, in call_subprocess raise error pipenv.patched.pip._internal.exceptions.InstallationSubprocessError: git clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amaz onaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 128 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 675, in main() File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 661, in main _main( File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 645, in _main resolve_packages( File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 612, in resolve_packages results, resolver = resolve( ^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 592, in resolve return resolve_deps( ^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 932, in resolve_deps results, hashes, internal_resolver = actually_resolve_deps( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 700, in actually_resolve_deps resolver.resolve() File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 457, in resolve raise ResolutionFailure(message=str(e)) pipenv.exceptions.ResolutionFailure: [31m[1mERROR[0m: [33mgit clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amaz onaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 128[0m ✘ Locking Failed! ⠋ Locking packages... Traceback (most recent call last): File "/home/me/.local/bin/pipenv", line 8, in sys.exit(cli()) ^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main return super().main(*args, **kwargs, windows_expand_args=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func return ctx.invoke(f, obj, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/command.py", line 209, in install do_install( File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 93, in do_install do_init( File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 653, in do_init do_lock( File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/lock.py", line 66, in do_lock venv_resolve_deps( File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps c = resolve(cmd, st, project=project) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 737, in resolve raise RuntimeError("Failed to lock Pipfile.lock!") RuntimeError: Failed to lock Pipfile.lock! ```

Steps to replicate

first

  1. create a private aws codecommit repo with a minimal pyproject.toml, as follows, and setup a user with git credentials (perhaps a private github repo would also work, but haven't tried that yet)

    [build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.0.1"
  2. in a bash terminal on the local system, create env variables USERNAME and PASSWORD (e.g. using read -s) with the corresponding values (url-encoded)
  3. create an empty dir, locally, and use pipenv to install the repo, in editable mode (not sure if that matters), as in: 
    pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"
  4. observe that this installation succeeds

then

  1. in the same terminal, create another empty dir

  2. copy the Pipfile, generated above, into the new dir, and replace the actual username and password in the file by the corresponding env variables, as in:

    [[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
mypackage = {editable = true, ref = "main", git = "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage"}

[dev-packages]

[requires]
python_version = "3.11"
  3. run pipenv install
  4. observe the status 403 error
$ pipenv --support Pipenv version: `'2024.0.1'` Pipenv location: `'/home/me/.local/lib/python3.11/site-packages/pipenv'` Python location: `'/home/me/.pyenv/versions/3.11.6/bin/python3.11'` OS Name: `'posix'` User pip version: `'24.0'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.11.6', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '6.5.0-41-generic', 'platform_system': 'Linux', 'platform_version': '#41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 3 ' '11:32:55 UTC 2', 'python_full_version': '3.11.6', 'python_version': '3.11', 'sys_platform': 'linux'} ``` System environment variables: - `SHELL` - `SESSION_MANAGER` - `QT_ACCESSIBILITY` - `PIPENV_VENV_IN_PROJECT` - `COLORTERM` - `PYENV_SHELL` - `XDG_CONFIG_DIRS` - `SSH_AGENT_LAUNCHER` - `XDG_MENU_PREFIX` - `GNOME_DESKTOP_SESSION_ID` - `LANGUAGE` - `LC_ADDRESS` - `GNOME_SHELL_SESSION_MODE` - `LC_NAME` - `SSH_AUTH_SOCK` - `GIT_PS1_SHOWDIRTYSTATE` - `XMODIFIERS` - `DESKTOP_SESSION` - `LC_MONETARY` - `GTK_MODULES` - `PWD` - `LOGNAME` - `XDG_SESSION_DESKTOP` - `XDG_SESSION_TYPE` - `SYSTEMD_EXEC_PID` - `XAUTHORITY` - `HOME` - `USERNAME` - `IM_CONFIG_PHASE` - `LC_PAPER` - `LANG` - `LS_COLORS` - `XDG_CURRENT_DESKTOP` - `VTE_VERSION` - `WAYLAND_DISPLAY` - `GNOME_TERMINAL_SCREEN` - `GNOME_SETUP_DISPLAY` - `LESSCLOSE` - `XDG_SESSION_CLASS` - `TERM` - `LC_IDENTIFICATION` - `LESSOPEN` - `USER` - `GNOME_TERMINAL_SERVICE` - `DISPLAY` - `SHLVL` - `LC_TELEPHONE` - `QT_IM_MODULE` - `LC_MEASUREMENT` - `PAPERSIZE` - `XDG_RUNTIME_DIR` - `PYENV_ROOT` - `LC_TIME` - `XDG_DATA_DIRS` - `PATH` - `GDMSESSION` - `DBUS_SESSION_BUS_ADDRESS` - `LC_NUMERIC` - `_` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: - `PIPENV_VENV_IN_PROJECT`: `1` Debug–specific environment variables: - `PATH`: `/home/me/.pyenv/versions/3.9.18/bin:/home/me/.pyenv/versions/3.8.18/bin:/home/me/.pyenv/versions/3.11.6/bin:/home/me/.pyenv/versions/3.8.13/bin:/home/me/.pyenv/shims:/home/me/.npm-global/bin:/home/me/.local/bin:/home/me/.pyenv/bin:/home/me/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin` - `SHELL`: `/bin/bash` - `LANG`: `en_CA.UTF-8` - `PWD`: `/home/me` ---------------------------
matteius commented 1 month ago

I am wondering if anyone can setup a sample private repo for testing this case -- I'd like to help improve the code around it.

dennisvang commented 1 month ago

@matteius I've tried to reproduce this using one of my private github repos, because those are easier to set up. However, in this case pipenv already fails at replication step 3. mentioned above (non-editable).

For example, pip succeeds (with github username and fine-grained personal access token as credentials.)

pip install "git+https://${USERNAME}:${PASSWORD}@github.com/myusername/my-private-repo.git"

but pipenv, with the same command, same terminal, same credentials

pipenv install "git+https://${USERNAME}:${PASSWORD}@github.com/myusername/my-private-repo.git"

fails with 

...
Added my-private-repo to Pipfile's [packages] ...
✔ Installation Succeeded
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (13a959)...
[pipenv.exceptions.InstallError]: Collecting my-private-repo@ git+https://myusername:****@0888...b155 (from -r /tmp/pipenv-hhg4plkm-requirements/pipenv-vmlg4kcl-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Cloning https://myusername:****@0888...b155 to /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]: Running command git clone --filter=blob:none --quiet 'https://myusername:****@0888...b155' /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]:   fatal: unable to access 'https://0888...b155/': Could not resolve host: 0888...b155
...
ERROR: Couldn't install package: {}
Package installation failed...

It looks unrelated to the present issue. Should I open a new issue for this?

matteius commented 6 days ago

@dennisvang Could you test this report against this branch with I believe does better with the VCS env variables: https://github.com/pypa/pipenv/pull/6242