search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.73k stars 1.86k forks source link

Pipfile with private index AND optional dependency doesn't work #6187

Open scastria opened 1 week ago

scastria commented 1 week ago

I have been using pipenv 2022.9.21 for a long time and decided to try the latest and greatest. The latest 2024.0.1 appears broken when it comes to locating packages in my private AWS CodeArtifact repo.

Issue description

pipenv install fails with this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
"my_utils[markdown]" = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

I thought it might be a failure in resolving the env variable referenced in the alternate source index called data_engineering so I hardcoded it in the Pipfile. No change.

Expected result

my_utils package is downloaded successfully

Actual result

Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠏ Locking packages...False
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement my_utils==0.1.391 (from versions: none)
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 645, in _main
[ResolutionFailure]:       resolve_packages(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 612, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/resolver.py", line 592, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 932, in resolve_deps
[ResolutionFailure]:       results, hashes, internal_resolver = actually_resolve_deps(
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 700, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 457, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv run pip install <requirement_name> to bypass this mechanism, then run $ pipenv graph to inspect the versions actually installed in the virtualenv.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for my_utils==0.1.391

Traceback (most recent call last):
  File "/opt/homebrew/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/install.py", line 653, in do_init
    do_lock(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

pipenv install

$ pipenv --support Pipenv version: `'2024.0.1'` Pipenv location: `'/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/lib/python3.12/site-packages/pipenv'` Python location: `'/opt/homebrew/Cellar/pipenv/2024.0.1/libexec/bin/python'` OS Name: `'posix'` User pip version: `'24.0'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.12.4', 'os_name': 'posix', 'platform_machine': 'arm64', 'platform_python_implementation': 'CPython', 'platform_release': '23.5.0', 'platform_system': 'Darwin', 'platform_version': 'Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT ' '2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000', 'python_full_version': '3.12.4', 'python_version': '3.12', 'sys_platform': 'darwin'} ``` System environment variables: - `MANPATH` - `TERM_PROGRAM` - `SHELL` - `TERM` - `HISTSIZE` - `HOMEBREW_REPOSITORY` - `TMPDIR` - `GOBIN` - `TERM_PROGRAM_VERSION` - `ADOPS_PAT` - `OLDPWD` - `TERM_SESSION_ID` - `TF_VAR_postman_localhost_api_key` - `FORGE_EMAIL` - `USER` - `LDAP_USERNAME` - `CODEARTIFACT_AUTH_TOKEN` - `SSH_AUTH_SOCK` - `PATH` - `__CFBundleIdentifier` - `PWD` - `JAVA_HOME` - `LANG` - `LDAP_PASSWORD` - `FORGE_API_TOKEN` - `XPC_FLAGS` - `PS1` - `ENVIRONMENT` - `XPC_SERVICE_NAME` - `SHLVL` - `HOME` - `SHELL_SESSION_HISTORY` - `HOMEBREW_PREFIX` - `LOGNAME` - `LOCAL_MYSQL_ROOT_PASSWORD` - `GOPATH` - `INFOPATH` - `HOMEBREW_CELLAR` - `BITBUCKET_PAT` - `HISTTIMEFORMAT` - `_` - `__CF_USER_TEXT_ENCODING` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/opt/homebrew/bin:/opt/homebrew/sbin:/Users/shawncastrianni/.go/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/opt/liquibase` - `SHELL`: `/bin/bash` - `LANG`: `en_US.UTF-8` - `PWD`: `/Users/shawncastrianni/GIT/data_api_one` --------------------------- Contents of `Pipfile` ('/Users/shawncastrianni/GIT/data_api_one/Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [[source]] url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/" verify_ssl = true name = "data_engineering" [packages] "my_utils[markdown]" = {version="==0.1.391", index="data_engineering"} [dev-packages] [requires] python_version = "3.10" ```
scastria commented 1 week ago

I also tried verify_ssl = false, but no change

scastria commented 1 week ago

I discovered a BIG clue to this. It seems to be related to the optional dependency of the my_utils package called markdown. If I remove mention of the optional dependency like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
my_utils = {version="==0.1.391", index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"

then the pipenv install command worked. It seems the optional dependency causes a problem. The optional dependency IS VALID as pipenv 2022.9.21 works just fine.

scastria commented 1 week ago

Another test is that I removed the my_utils dependency completely from the Pipfile and instead specified ONLY "redis[hiredis]" = "==4.6.0" and that worked too. So optional dependencies work against PyPI, but they do NOT work against private indicies.

scastria commented 1 week ago

I wasn't aware of the extras keyword in Pipfile. If I change the Pipfile to use extras, it DOES work. So I guess now the question is should the extras syntax be required or can the other syntax without extras be used?

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://aws:${CODEARTIFACT_AUTH_TOKEN}@data-engineering-123456789.d.codeartifact.us-west-2.amazonaws.com/pypi/data_engineering/simple/"
verify_ssl = true
name = "data_engineering"

[packages]
my_utils = {version="==0.1.391", extras=["markdown"], index="data_engineering"}

[dev-packages]

[requires]
python_version = "3.10"