search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.46k stars 3k forks source link

pip should not cache large files in /tmp or TMPDIR #12868

Open lee-b opened 1 month ago

lee-b commented 1 month ago

Description

Pip violates system specifications, and therefore essentially only works by accident right now, when it "gets lucky" with packages being small enough for /tmp.

This becomes a serious problem when working with larger pip installations, such as for vllm, pytorch with cuda acceleration, etc.

5816 was closed with the advice to ensure that /tmp is large enough. #4462 was on the same topic and also closed without action. Pip also blames the user/system, with this output:

OSError: [Errno 28] No space left on device
...
note: This error originates from a subprocess, and is likely not a problem with pip.

However, these analyses are NOT correct.

If one reads the Linux file-hierarchy (7) man page specification (i.e., runs man 7 file-hierarchy) on Linux (Debian 12, at least), it states:

       /tmp/
           The place for small temporary files. This directory is usually mounted as a "tmpfs" instance, and should
           hence not be used for larger files. (Use /var/tmp/ for larger files.)

This document also refers readers to:

https://systemd.io/TEMPORARY_DIRECTORIES/

Which similarly states:

/tmp/ and /var/tmp/ are two world-writable directories Linux systems provide for temporary files. The former is typically on tmpfs and thus backed by RAM/swap, and flushed out on each reboot. The latter is typically a proper, persistent file system, and thus backed by physical storage. This means:

    /tmp/ should be used for smaller, size-bounded files only; /var/tmp/ should be used for everything else.

Moreover, the data in question seems to be cached data, not normal temporary files, and should therefore go in /var/cache (probably only if daemon is writing it, I believe), or into the XDG cache directory (e.g., ~/.cache/pip/).

Expected behavior

Pip should follow all relevant specifications when creating files, rather than putting large files in the wrong place and overloading filesystems that are not intended for large files.

pip version

24.0

Python version

3.12.2

OS

Debian 12

How to Reproduce

bin/pip3 install vllm when /tmp has 1.7GB available.

Output

Collecting vllm
  Using cached vllm-0.5.2-cp38-abi3-manylinux1_x86_64.whl.metadata (1.8 kB)
Collecting aiohttp (from vllm)
  Downloading aiohttp-3.9.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.5 kB)
Collecting cmake>=3.21 (from vllm)
  Using cached cmake-3.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
Collecting fastapi (from vllm)
  Using cached fastapi-0.111.1-py3-none-any.whl.metadata (26 kB)
Collecting filelock>=3.10.4 (from vllm)
  Using cached filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
Collecting lm-format-enforcer==0.10.3 (from vllm)
  Using cached lm_format_enforcer-0.10.3-py3-none-any.whl.metadata (16 kB)
Collecting ninja (from vllm)
  Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Collecting numpy<2.0.0 (from vllm)
  Using cached numpy-1.26.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting nvidia-ml-py (from vllm)
  Using cached nvidia_ml_py-12.555.43-py3-none-any.whl.metadata (8.6 kB)
Collecting openai (from vllm)
  Using cached openai-1.36.1-py3-none-any.whl.metadata (22 kB)
Collecting outlines<0.1,>=0.0.43 (from vllm)
  Using cached outlines-0.0.46-py3-none-any.whl.metadata (15 kB)
Collecting pillow (from vllm)
  Using cached pillow-10.4.0-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (9.2 kB)
Collecting prometheus-fastapi-instrumentator>=7.0.0 (from vllm)
  Using cached prometheus_fastapi_instrumentator-7.0.0-py3-none-any.whl.metadata (13 kB)
Collecting prometheus-client>=0.18.0 (from vllm)
  Using cached prometheus_client-0.20.0-py3-none-any.whl.metadata (1.8 kB)
Collecting psutil (from vllm)
  Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting py-cpuinfo (from vllm)
  Using cached py_cpuinfo-9.0.0-py3-none-any.whl.metadata (794 bytes)
Collecting pydantic>=2.0 (from vllm)
  Using cached pydantic-2.8.2-py3-none-any.whl.metadata (125 kB)
Collecting pyzmq (from vllm)
  Downloading pyzmq-26.0.3-cp312-cp312-manylinux_2_28_x86_64.whl.metadata (6.1 kB)
Collecting ray>=2.9 (from vllm)
  Downloading ray-2.32.0-cp312-cp312-manylinux2014_x86_64.whl.metadata (13 kB)
Collecting requests (from vllm)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting sentencepiece (from vllm)
  Downloading sentencepiece-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting tiktoken>=0.6.0 (from vllm)
  Downloading tiktoken-0.7.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting tokenizers>=0.19.1 (from vllm)
  Downloading tokenizers-0.19.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting torch==2.3.1 (from vllm)
  Downloading torch-2.3.1-cp312-cp312-manylinux1_x86_64.whl.metadata (26 kB)
Collecting torchvision==0.18.1 (from vllm)
  Downloading torchvision-0.18.1-cp312-cp312-manylinux1_x86_64.whl.metadata (6.6 kB)
Collecting tqdm (from vllm)
  Using cached tqdm-4.66.4-py3-none-any.whl.metadata (57 kB)
Collecting transformers>=4.42.4 (from vllm)
  Using cached transformers-4.42.4-py3-none-any.whl.metadata (43 kB)
Collecting typing-extensions (from vllm)
  Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting uvicorn[standard] (from vllm)
  Using cached uvicorn-0.30.3-py3-none-any.whl.metadata (6.5 kB)
INFO: pip is looking at multiple versions of vllm to determine which version is compatible with other requirements. This could take a while.
Collecting vllm
  Downloading vllm-0.5.1.tar.gz (790 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 790.6/790.6 kB 7.0 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [64 lines of output]
      Collecting cmake>=3.21
        Using cached cmake-3.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.1 kB)
      Collecting ninja
        Using cached ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
      Collecting packaging
        Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
      Collecting setuptools>=49.4.0
        Downloading setuptools-71.0.4-py3-none-any.whl.metadata (6.5 kB)
      Collecting torch==2.3.0
        Downloading torch-2.3.0-cp312-cp312-manylinux1_x86_64.whl.metadata (26 kB)
      Collecting wheel
        Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
      Collecting filelock (from torch==2.3.0)
        Using cached filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
      Collecting typing-extensions>=4.8.0 (from torch==2.3.0)
        Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
      Collecting sympy (from torch==2.3.0)
        Using cached sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
      Collecting networkx (from torch==2.3.0)
        Using cached networkx-3.3-py3-none-any.whl.metadata (5.1 kB)
      Collecting jinja2 (from torch==2.3.0)
        Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
      Collecting fsspec (from torch==2.3.0)
        Using cached fsspec-2024.6.1-py3-none-any.whl.metadata (11 kB)
      Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.3.0)
        Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.3.0)
        Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.3.0)
        Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.3.0)
        Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
      Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.3.0)
        Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.3.0)
        Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
      Collecting nvidia-nccl-cu12==2.20.5 (from torch==2.3.0)
        Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
      Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.3.0)
        Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
      Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.0)
        Using cached nvidia_nvjitlink_cu12-12.5.82-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
      Collecting MarkupSafe>=2.0 (from jinja2->torch==2.3.0)
        Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
      Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.3.0)
        Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
      Downloading torch-2.3.0-cp312-cp312-manylinux1_x86_64.whl (779.1 MB)
         ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 779.1/779.1 MB 5.3 MB/s eta 0:00:00
      Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
      Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
      Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
      Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
      Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
      ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device

      [notice] A new release of pip is available: 24.0 -> 24.1.2
      [notice] To update, run: /mnt/nvme1/home/lb/.local/.venvs/vllm/bin/python3 -m pip install --upgrade pip
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

Code of Conduct

pfmoore commented 1 month ago

The data is temporary files (it's the unpacked source of vllm-0.5.1.tar.gz which appears to need a lot of space to build). Pip respects the standard $TEMP/$TMPDIR directories, which as far as I am aware are not defined as being limited to "small files only". That would suggest to me that $TEMP should be set to /var/tmp rather than to /tmp on systems that limit the space available to /tmp. But I'm not a Unix expert, so I don't know the nuances of how that environment variable is intended to be set if the filesystem standards define two "levels" of temporary filesystem space.

In fact, pip simply uses Python's standard temporary file management functions, which are also not limited to "small files only". So I would suggest that if you disagree with setting $TEMP to /var/tmp, you'd need to take it up with the Python project.

lee-b commented 1 month ago

which as far as I am aware are not defined as being limited to "small files only".

I mean, I literally gave you the definitions, "The place for small temporary files", "/tmp/ should be used for smaller, size-bounded files only; /var/tmp/ should be used for everything else."

pfmoore commented 1 month ago

I mean, I literally gave you the definitions

For the filesystem locations, yes. I'm not disputing that. But why is $TEMP set to /tmp if that location is only for small files? I've never seen any documentation that states that programs should not store large files in the directory pointed at by $TEMP. If there is such documentation, then:

  1. Please provide a link.
  2. Please raise this with the CPython project, as they absolutely do not document that the tempfile module must only be used for small files (see here).

But realistically, I would say that you should simply set $TEMP to /var/tmp, and that would solve the issue for you.

notatallshaw commented 1 month ago

Just because I was curious about the spec, here is some reading:

uranusjr commented 1 month ago

While systemd defines the two temp directories as you mentioned, I don’t see the distinction being made anywhere by systems that actually create the two directories for software to use. Even the reference they ultimate link to, https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html, has nothing to distinguish vetween /tmp and /var/tmp, so at best the entire situation is [citation needed] and boarderline systemd is just making rules up.

pradyunsg commented 1 month ago

Yea, it sounds like the situation is the exact same as it was back when I'd posted https://github.com/pypa/pip/issues/5816#issuecomment-587302775.

I think one thing called out in OP that's worth figuring out how to improve is:

blames the user/system, with this output:

note: This error originates from a subprocess, and is likely not a problem with pip.

I think it's worth improving this line removing it when pip calls itself in a subprocess (the error will contain this message at the relevant place).

5816 was closed with the advice to ensure that /tmp is large enough.

As the person who wrote the closing comment, this is not what I wrote or meant. I'd explicitly referenced $TMPDIR for a reason in that comment. :)

lee-b commented 1 month ago

Note the reasoning and context (of Solaris etc.) given here, as well as the advice to fixapplications that write large files to use /var/tmp:

https://fedoraproject.org/wiki/Features/tmp-on-tmpfs#Comments_and_Discussion

pfmoore commented 1 month ago

I’ll repeat my comment from above - there is nothing actionable for pip here (except possibly an improved error message or documentation, as noted by @pradyunsg). We aren’t going to stop using the stdlib functionality, so the only realistic approaches here are a stdlib change (unlikely, IMO) or a user config change (the recommended approach)

eli-schwartz commented 1 month ago

While systemd defines the two temp directories as you mentioned, I don’t see the distinction being made anywhere by systems that actually create the two directories for software to use. Even the reference they ultimate link to, https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html, has nothing to distinguish vetween /tmp and /var/tmp, so at best the entire situation is [citation needed] and boarderline systemd is just making rules up.

If you reread the FHS citation, you will notice that there is in fact something to distinguish between /tmp and /var/tmp. That thing is, that /var/tmp is specified to survive system reboots (and therefore logically cannot be on a RAM-backed tmpfs as it will not survive a reboot).

If you assume a bunch of things:

then /var/tmp may have much more available space than /tmp. It is unlikely to have less space.

That is probably the reason why systemd "requires" the rules it has made up -- because systemd likes to assume things based on what its authors would choose to do, and then rule out all other possible scenarios.

But, the systemd advice is a very very bad idea to follow if you have 64gb of RAM and your /var/tmp has a quota / dedicated partition and can "only" store 8gb on it -- you'll get 4x less space in /var/tmp than you'll get in /tmp, because /tmp usually allows half your available RAM...

eli-schwartz commented 1 month ago

It would be very, very wrong for pip to declare that its work-in-progress compiled packages are de-facto "temporary files or directories that are preserved between system reboots", unless pip has changed its approach quite a bit...

eli-schwartz commented 1 month ago

The actual, correct solution here is for someone to write a spec such as https://devmanual.gentoo.org/eclass-reference/check-reqs.eclass/index.html

It would allow python source code that is automatically built into a wheel, to declare in advance the amount of space it estimates it will probably require, so that tools such as pip can query the underlying filesystem for the directory that has been created by tempfile and check if there is, in fact, enough free space for the build to succeed.

pip could then error out before doing anything at all, with a clever error message such as:

ERROR: vllm asked for 3GB of temporary storage, but this is not available. Cannot build vllm. Please set the $TMPDIR environment variable to some location that has the required disk space, and try again.
morotti commented 1 month ago

I'm having a look through the issue, as I've experienced it quite a bit too. It's not a pip issue, it's an issue with the user system being misconfigured on setup and it's highly specific to Linux.

The user's system was misconfigured with too small partitions (I've have endless problems in some companies with infra teams provisioning ridiculously small partitions on VMs). The system needs to be resized.

There may be a broader issue with some Linux distributions defaulting to smaller /tmp in memory, since recently, which would be an issue to report to distributions.

There was a similar issue with extremely small disks around 2015 when AWS became popular. VM images had 10GB disks to fit with the free tier, which created a lot of problems with running out of disks all the time and everything crashing. Eventually the provisioning tools became able to resize VM on creation and people learned to use it.

Users can run df -h to see partition size and resize anything partition that's single digit. Users can set TMPDIR to specify a different location to build packages.

@eli-schwartz what is the output of df -h for you? Is it a personal machine you setup yourself or a machine at work setup by someone else?

eli-schwartz commented 1 month ago

@morotti I think you misunderstood my stake in this. :)

I saw a recent issue where someone was spreading FUD about systemd and claiming that pip isn't "compliant with systemd requirements". I like systemd quite well as it happens, I simply don't think that this one specific rationale in the systemd documentation holds water.

In fact, I don't use pip to install large packages. I generally regard the need to compile C/C++/Fortran software as a sign that pip is the wrong tool for the job and you should be using conda or a linux distro's prebuilt packages.

But I don't think that pip is doing the wrong thing here and I don't think that pip's developers should have to be lectured about how they're doing the wrong thing because systemd.

...

All that being said:

I think it's false to say that /tmp is in memory, I don't think it was in memory historically. Most computers in the 2010s had 1-4 GB of memory (single digit) and there wasn't enough memory to hold anything.

No one claimed it is? The claim was that it "usually" is -- and this claim is correct in the sense that it is a popular mechanism for system distributors, and also, if your system happens to use systemd, systemd will mount /tmp as a tmpfs backed by RAM automatically unless you go out of your way to change the defaults by masking it.

1-4 GB of memory is certainly enough memory to hold plenty of things regardless. Especially if you try to optimize for the average desktop user, who doesn't use /tmp interactively but has various programs writing short-lived temporary files there, which on average don't go above 100mb.

I don't recall /tmp being being limited to single digit size and being in memory until recently. It seems to me the issue was introduced recently.

It's not limited to single digit size, and it's not recent. Fedora has been doing it since 2012, systemd provided the configs for it since 2010. Early on, Debian disabled this based on initial feedback and hasn't really revisited the topic in 12 years -- but now they're finally changing and will do the same.

tensorflow/torch/cuda require 10+GB to install and use, they don't play well with small partitions. Users simply won't be able to use them if they have ridiculously small disks.

This bug report was about COMPILING packages, not using them.

Due to pip's policy of COMPILING packages in tempfile.mkdtemp() using an isolated venv, if you want to COMPILE a package that requires cuda at COMPILATION time, you must have enough space to install the files into a temporary virtualenv which by default is in /tmp.

No usage is implied. And if you can get past that hurdle and install it, you are not installing your workspace to $TMPDIR so the question becomes moot.

I'm having a look through the issue, as I've experienced it quite a bit too. It's not a pip issue, it's an issue with the user system being misconfigured on setup and it's highly specific to Linux.

The user's system was misconfigured with too small partitions (I've have endless problems in some companies with infra teams provisioning ridiculously small partitions on VMs). The system needs to be resized.

There may be a broader issue with some Linux distributions defaulting to smaller /tmp in memory, since recently, which would be an issue to report to distributions.

It is not specific to Linux, there is an entire industry of Windows software for creating ramdisks and moving %TEMP% to it. Solaris started making /tmp a tmpfs back in 1994, long before it became popular on Linux.

eli-schwartz commented 1 month ago

Users can set TMPDIR to specify a different location to build packages.

Correct, this has been repeatedly suggested. :)