search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.9k stars 1.87k forks source link

Locking fails when using `--extra-pip-args="--proxy=` #6003

Open Helyux opened 1 year ago

Helyux commented 1 year ago

Issue description

Can't install a package behind a proxy using the --extra-pip-args flag. The locking fails with pipenv.exceptions.ResolutionFailure.

following pip command works: pip install --proxy 127.0.0.1:3128 numpy following pipenv command doesn't: pipenv install --extra-pip-args="--proxy=127.0.0.1:3128" numpy

Important detail: I'm utilizing a Px proxy to bypass the actual NTLM proxy. The 407 errors listed below are the results I encounter when attempting a basic pipenv install numpy. That's why I suspect that the Px proxy I provided isn't being utilized.

Expected result

Package gets added to pipfile and the pipfile.lock is updated.

Actual result

C:\Users\helyux\Desktop\Aiden>pipenv install --verbose --extra-pip-args="--proxy=127.0.0.1:3128" numpy
Installing numpy...
Resolving numpy...
Added numpy to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock (e4eef2) out of date, updating to (a6e810)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('numpy'), None)
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement numpy (from versions: none)
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 174, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pipenv.patched.pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('numpy')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\resolution\resolvelib\resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 399, in resolve
    raise ResolutionImpossible(e.criterion.information)
pipenv.patched.pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('numpy'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 440, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\resolution\resolvelib\resolver.py", line 104, in resolve
    raise error from e
pipenv.patched.pip._internal.exceptions.DistributionNotFound: No matching distribution found for numpy
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 675, in <module>
    main()
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 661, in main
    _main(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 645, in _main
    resolve_packages(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 592, in resolve
    return resolve_deps(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 908, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 681, in actually_resolve_deps
    resolver.resolve()
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 442, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for numpy
Locking Failed!
[   =] Locking...
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python310\Scripts\pipenv.exe\__main__.py", line 7, in <module>
    # when invoked as python -m pip <command>
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 297, in do_install
    raise e
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 281, in do_install
    do_init(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 648, in do_init
    do_lock(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 849, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 718, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

Create a new project pipenv install Install a package passing a proxy pipenv install --extra-pip-args="--proxy=127.0.0.1:3128" numpy

$ pipenv --support Pipenv version: `'2023.10.24'` Pipenv location: `'C:\\Users\\helyux\\AppData\\Roaming\\Python\\Python310\\site-packages\\pipenv'` Python location: `'C:\\Program Files\\Python310\\python.exe'` OS Name: `'nt'` User pip version: `'23.3.1'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.10.6', 'os_name': 'nt', 'platform_machine': 'AMD64', 'platform_python_implementation': 'CPython', 'platform_release': '10', 'platform_system': 'Windows', 'platform_version': '10.0.22000', 'python_full_version': '3.10.6', 'python_version': '3.10', 'sys_platform': 'win32'} ``` System environment variables: - `ALLUSERSPROFILE` - `APPDATA` - `COMMONPROGRAMFILES` - `COMMONPROGRAMFILES(X86)` - `COMMONPROGRAMW6432` - `COMPUTERNAME` - `COMSPEC` - `DRIVERDATA` - `FPS_BROWSER_APP_PROFILE_STRING` - `FPS_BROWSER_USER_PROFILE_STRING` - `GOPATH` - `HOME` - `HOMEDRIVE` - `HOMEPATH` - `HOMESHARE` - `JAVA_HOME` - `LOCALAPPDATA` - `LOGONSERVER` - `MAGICK_HOME` - `NUMBER_OF_PROCESSORS` - `ONEDRIVE` - `OS` - `PATH` - `PATHEXT` - `PROCESSOR_ARCHITECTURE` - `PROCESSOR_IDENTIFIER` - `PROCESSOR_LEVEL` - `PROCESSOR_REVISION` - `PROGRAMDATA` - `PROGRAMFILES` - `PROGRAMFILES(X86)` - `PROGRAMW6432` - `PROMPT` - `PSMODULEPATH` - `PUBLIC` - `PYCHARM COMMUNITY EDITION` - `SESSIONNAME` - `SYSTEMDRIVE` - `SYSTEMROOT` - `TEMP` - `TMP` - `USERDNSDOMAIN` - `USERDOMAIN` - `USERDOMAIN_ROAMINGPROFILE` - `USERNAME` - `USERPROFILE` - `WINDIR` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenvûspecific environment variables: Debugûspecific environment variables: --------------------------- Contents of `Pipfile` ('C:\\Users\\helyux\\Desktop\\Aiden\\Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] [dev-packages] [requires] python_version = "3.10" ``` Contents of `Pipfile.lock` ('C:\\Users\\helyux\\Desktop\\Aiden\\Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "fedbd2ab7afd84cf16f128af0619749267b62277b4cb6989ef16d4bef6e4eef2" }, "pipfile-spec": 6, "requires": { "python_version": "3.10" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": {}, "develop": {} } ```
matteius commented 1 year ago

Have you tried passing this just as a regular --index url option or --pypi-mirror url? I think with the way we have customized the resolver, what you are trying to do wouldn't work with extra pip args, as you point out.

Helyux commented 1 year ago

Hello Matteius, I'll give it a try on Monday. Thank you for the suggestion. I'm curious about why it's included as an example in the documentation then. Am I perhaps misunderstanding something? advanced/

matteius commented 1 year ago

The doc example is for sync which is skipping a lock resolution phase. It turns out extra pip args were never passed to the resolver, but I have a PR out that does this and I am looking for feedback on the various use cases. https://github.com/pypa/pipenv/pull/6006

Helyux commented 1 year ago

Passing the proxy address to --index (pipenv install --index 127.0.0.1:3128 numpy) resulted in the same 407's as without any args. Passing to --pypi-mirror resulted in Error: Invalid value for '--pypi-mirror': Invalid PyPI mirror URL: 127.0.0.1:3128

Sadly i don't know how to update my pipenv.exe to your patched version, if you can give me a pointer on that i'm more then happy to test.

matteius commented 1 year ago

@Helyux Does your proxy require authentication parameters? https://stackoverflow.com/questions/46466241/pip-install-failing-with-407-proxy-authentication-required

Helyux commented 1 year ago

It does but that's what the px proxy is for, it automatically handles the authentication, see my initial comment.

matteius commented 1 year ago

@Helyux to try the branch: pip uninstall pipenv --yes pip install git+https://github.com/pypa/pipenv.git@issue-6001-extra-pip-args

matteius commented 3 weeks ago

@Helyux I'd be curious what your result with this is on 2024.3.0?

Helyux commented 3 weeks ago

Hi @matteius, sadly i'm unable to test & reproduce now due to new corporate restrictions. Feel free to close the issue if you see no further need for it.