search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.83k stars 1.87k forks source link

pipenv cannot install pytorch for cuda 11 even though pip can #5996

Closed jeffhhk closed 10 months ago

jeffhhk commented 11 months ago

Issue description

pipenv cannot install pytorch for cuda 11 even though pip can.

pytorch for cuda 11 is distributed in an alternate repository location.

Expected result

I would expect pipenv to be able to parse and install packages from the alternate repository location.

Actual result

pipenv.patched.pip._internal.exceptions.InstallationError: Cannot determine archive format of /tmp/tmpglf6gk37

I do not see any filed issues with a very similar error.

I reinstalled pipenv and made a test empty Pipfile to make sure the problem description was minimal.

Steps to replicate

pipenv --version
    pipenv, version 2023.10.24
cat Pipfile 
    [[source]]
    url = "https://pypi.org/simple"
    verify_ssl = true
    name = "pypi"

    [packages]

    [dev-packages]

    [requires]
    python_version = "3.10"
pipenv shell
pipenv install https://download.pytorch.org/whl/cu118 torch
    Installing https://download.pytorch.org/whl/cu118...
    Resolving https://download.pytorch.org/whl/cu118...
    ⠦ Installing None...INFO:pipenv.patched.pip._internal.network.download:Downloading https://download.pytorch.org/whl/cu118 (1.7 kB)
    CRITICAL:pipenv.patched.pip._internal.utils.unpacking:Cannot unpack file /tmp/pip-unpack-hhd4643m/cu118.html (downloaded from /tmp/tmpglf6gk37, content-type: text/html); cannot detect archive format
    Traceback (most recent call last):
      File "/home/jeff/.local/bin/pipenv", line 8, in <module>
        sys.exit(cli())
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
        return self.main(*args, **kwargs)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/cli/options.py", line 58, in main
        return super().main(*args, **kwargs, windows_expand_args=False)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1078, in main
        rv = self.invoke(ctx)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
        return _process_result(sub_ctx.command.invoke(sub_ctx))
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
        return ctx.invoke(self.callback, **ctx.params)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
        return __callback(*args, **kwargs)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
        return ctx.invoke(f, obj, *args, **kwargs)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
        return __callback(*args, **kwargs)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/cli/command.py", line 209, in install
        do_install(
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/routines/install.py", line 255, in do_install
        added, cat, normalized_name = project.add_package_to_pipfile(
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/project.py", line 1214, in add_package_to_pipfile
        name, normalized_name, entry = self.generate_package_pipfile_entry(
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/project.py", line 1158, in generate_package_pipfile_entry
        req_name = determine_package_name(package)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/utils/dependencies.py", line 744, in determine_package_name
        local_file = unpack_url(
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/utils/requirementslib.py", line 713, in unpack_url
        unpack_file(file.path, location, file.content_type)
      File "/home/jeff/.local/lib/python3.8/site-packages/pipenv/patched/pip/_internal/utils/unpacking.py", line 257, in unpack_file
        raise InstallationError(f"Cannot determine archive format of {location}")
    pipenv.patched.pip._internal.exceptions.InstallationError: Cannot determine archive format of /tmp/tmpglf6gk37

yet:

pip install torch --index-url https://download.pytorch.org/whl/cu118
    Looking in indexes: https://download.pytorch.org/whl/cu118
    Collecting torch
      Downloading https://download.pytorch.org/whl/cu118/torch-2.1.0%2Bcu118-cp310-cp310-linux_x86_64.whl (2325.9 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 GB 2.2 MB/s eta 0:00:00
    Collecting filelock (from torch)
      Downloading https://download.pytorch.org/whl/filelock-3.9.0-py3-none-any.whl (9.7 kB)
    Collecting typing-extensions (from torch)
      Downloading https://download.pytorch.org/whl/typing_extensions-4.4.0-py3-none-any.whl (26 kB)
    Collecting sympy (from torch)
      Downloading https://download.pytorch.org/whl/sympy-1.12-py3-none-any.whl (5.7 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 44.7 MB/s eta 0:00:00
    Collecting networkx (from torch)
      Downloading https://download.pytorch.org/whl/networkx-3.0-py3-none-any.whl (2.0 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 54.9 MB/s eta 0:00:00
    Collecting jinja2 (from torch)
      Downloading https://download.pytorch.org/whl/Jinja2-3.1.2-py3-none-any.whl (133 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 13.7 MB/s eta 0:00:00
    Collecting fsspec (from torch)
      Downloading https://download.pytorch.org/whl/fsspec-2023.4.0-py3-none-any.whl (153 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 154.0/154.0 kB 17.6 MB/s eta 0:00:00
    Collecting triton==2.1.0 (from torch)
      Downloading https://download.pytorch.org/whl/triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 10.5 MB/s eta 0:00:00
    Collecting MarkupSafe>=2.0 (from jinja2->torch)
      Downloading https://download.pytorch.org/whl/MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
    Collecting mpmath>=0.19 (from sympy->torch)
      Downloading https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl (536 kB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 20.0 MB/s eta 0:00:00
    Installing collected packages: mpmath, typing-extensions, sympy, networkx, MarkupSafe, fsspec, filelock, triton, jinja2, torch
    Successfully installed MarkupSafe-2.1.2 filelock-3.9.0 fsspec-2023.4.0 jinja2-3.1.2 mpmath-1.3.0 networkx-3.0 sympy-1.12 torch-2.1.0+cu118 triton-2.1.0 typing-extensions-4.4.0

pipenv --support

$ pipenv --support Pipenv version: `'2023.10.24'` Pipenv location: `'/home/jeff/.local/lib/python3.8/site-packages/pipenv'` Python location: `'/usr/bin/python3'` OS Name: `'posix'` User pip version: `'23.3.1'` user Python installations found: PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.8.10', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '5.15.0-87-generic', 'platform_system': 'Linux', 'platform_version': '#97~20.04.1-Ubuntu SMP Thu Oct 5 08:25:28 UTC 2023', 'python_full_version': '3.8.10', 'python_version': '3.8', 'sys_platform': 'linux'} ``` System environment variables: - `SHELL` - `SESSION_MANAGER` - `CAML_LD_LIBRARY_PATH` - `PYENV_HOOK_PATH` - `QT_ACCESSIBILITY` - `OCAML_TOPLEVEL_PATH` - `COLORTERM` - `PYENV_SHELL` - `XDG_CONFIG_DIRS` - `XDG_MENU_PREFIX` - `GNOME_DESKTOP_SESSION_ID` - `SDKMAN_OLD_PWD` - `PIPENV_ACTIVE` - `LANGUAGE` - `SBT_HOME` - `MANDATORY_PATH` - `LC_ADDRESS` - `JAVA_HOME` - `GNOME_SHELL_SESSION_MODE` - `LC_NAME` - `SSH_AUTH_SOCK` - `SDKMAN_CANDIDATES_DIR` - `XMODIFIERS` - `DESKTOP_SESSION` - `LC_MONETARY` - `SSH_AGENT_PID` - `PYENV_VERSION` - `GTK_MODULES` - `DBUS_STARTER_BUS_TYPE` - `PWD` - `PYENV_VIRTUALENV_INIT` - `XDG_SESSION_DESKTOP` - `LOGNAME` - `XDG_SESSION_TYPE` - `MANPATH` - `GPG_AGENT_INFO` - `XAUTHORITY` - `OPAM_SWITCH_PREFIX` - `WINDOWPATH` - `NODENV_SHELL` - `PIP_PYTHON_PATH` - `HOME` - `USERNAME` - `IM_CONFIG_PHASE` - `LANG` - `LC_PAPER` - `LS_COLORS` - `XDG_CURRENT_DESKTOP` - `WASMTIME_HOME` - `VIRTUAL_ENV` - `VTE_VERSION` - `SDKMAN_VERSION` - `LEININGEN_HOME` - `INVOCATION_ID` - `MANAGERPID` - `PYENV_DIR` - `LESSCLOSE` - `XDG_SESSION_CLASS` - `LC_IDENTIFICATION` - `TERM` - `DEFAULTS_PATH` - `LESSOPEN` - `LIBVIRT_DEFAULT_URI` - `USER` - `PIP_DISABLE_PIP_VERSION_CHECK` - `SDKMAN_DIR` - `DISPLAY` - `SHLVL` - `LC_TELEPHONE` - `QT_IM_MODULE` - `LC_MEASUREMENT` - `SDKMAN_CANDIDATES_API` - `DBUS_STARTER_ADDRESS` - `PAPERSIZE` - `VIRTUAL_ENV_PROMPT` - `PYTHONDONTWRITEBYTECODE` - `TILIX_ID` - `XDG_RUNTIME_DIR` - `PYENV_ROOT` - `LC_TIME` - `JOURNAL_STREAM` - `XDG_DATA_DIRS` - `PATH` - `GDMSESSION` - `DBUS_SESSION_BUS_ADDRESS` - `SDKMAN_PLATFORM` - `LC_NUMERIC` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: - `PIPENV_ACTIVE`: `1` Debug–specific environment variables: - `PATH`: `/home/jeff/.pyenv/versions/jaxbuild/bin:/home/jeff/.pyenv/versions/3.9.8/bin:/home/jeff/.pyenv/versions/miniconda3-3.11-23.5.2-0/bin:/home/jeff/.pyenv/versions/global/bin:/home/jeff/.pyenv/versions/3.10.13/bin:/home/jeff/.pyenv/versions/linkml2/bin:/home/jeff/.pyenv/versions/miniconda3-4.7.12/bin:/home/jeff/.pyenv/versions/3.9.7/bin:/home/jeff/.pyenv/libexec:/home/jeff/.pyenv/plugins/python-build/bin:/home/jeff/.pyenv/plugins/pyenv-virtualenv/bin:/home/jeff/.pyenv/plugins/pyenv-update/bin:/home2TB/jeff/.local/share/virtualenvs/tmp_2023-10-26-kyHRqbtq/bin:/home/jeff/.elan/bin:/home2TB/jeff/.opam/default/bin:/home/jeff/.nodenv/shims:/home/jeff/.nodenv/bin:/home/jeff/.pyenv/plugins/pyenv-virtualenv/shims:/home/jeff/.pyenv/shims:/home/jeff/.wasmtime/bin:/home/jeff/.pyenv/libexec:/home/jeff/.pyenv/plugins/python-build/bin:/home/jeff/.pyenv/plugins/pyenv-virtualenv/bin:/home/jeff/.pyenv/plugins/pyenv-update/bin:/home/jeff/.elan/bin:/home/jeff/.sdkman/candidates/sbt/current/bin:/home/jeff/.sdkman/candidates/leiningen/current/bin:/home/jeff/.sdkman/candidates/java/current/bin:/home/jeff/.nodenv/shims:/home/jeff/.nodenv/bin:/home/jeff/.pyenv/plugins/pyenv-virtualenv/shims:/home/jeff/.wasmtime/bin:/home/jeff/.cargo/bin:/home/jeff/.local/bin:/home/jeff/bin:/home/jeff/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/racket-86-full/bin:/home/jeff/.cabal/bin:/home/jeff/.ghcup/bin:/usr/local/racket-86-full/bin` - `SHELL`: `/bin/bash` - `LANG`: `en_US.UTF-8` - `PWD`: `/home/jeff/tmp_2023-10-26` - `VIRTUAL_ENV`: `/home2TB/jeff/.local/share/virtualenvs/tmp_2023-10-26-kyHRqbtq` --------------------------- Contents of `Pipfile` ('/home/jeff/tmp_2023-10-26/Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] [dev-packages] [requires] python_version = "3.10" ```

Thank you for the effort toward improving the pip ecosystem.

matteius commented 11 months ago

You need to pass the index with --index so not: pipenv install https://download.pytorch.org/whl/cu118 torch but instead pipenv install torch --index https://download.pytorch.org/whl/cu118

jeffhhk commented 10 months ago

Wow, it works! I thought for sure I tried this . . .

The Pipfile afterwards:

    [[source]]
    url = "https://download.pytorch.org/whl/cu118"
    verify_ssl = true
    name = "downloadpytorch"

    [packages]
    . . .
    torch = {version = "*", index = "downloadpytorch"}

Many thanks!