search

mamba-org / mamba

The Fast Cross-Platform Package Manager
https://mamba.readthedocs.io
BSD 3-Clause "New" or "Revised" License
6.82k stars 351 forks source link

installation of pip packages fails if directory containing environment definition is read-only #2728

Open wholtz opened 1 year ago

wholtz commented 1 year ago

Troubleshooting docs

How did you install Mamba?

Micromamba

Search tried in issue tracker

Done

Latest version of Mamba

Tried in Conda?

I have this problem with Conda as well, without using Mamba

Describe your issue

When an environment definition file (environment.yaml) containing pip packages is used to create an environment, the installation will fail if the environment definition file is in a read-only directory. This is because mamba/micromamba/conda all attempt to create a temporary file in the same directory as the environment definition file. I have seen multiple instances of users getting tripped up by this in the mamba and micromamba-docker GitHub issue trackers (mamba #2720, micromamba #356).

I'm guessing mamba/micromamba just followed what conda did here, but I would much prefer that the directory for temporary files follows how mktemp selects where to place temporary files.

mamba info / micromamba info

$ docker run -it --rm -v $(pwd)/conda:/conda:ro  mambaorg/micromamba:latest micromamba info

       libmamba version : 1.4.9
     micromamba version : 1.4.9
           curl version : libcurl/7.88.1 OpenSSL/3.1.1 zlib/1.2.13 zstd/1.5.2 libssh2/1.11.0 nghttp2/1.52.0
     libarchive version : libarchive 3.6.2 zlib/1.2.13 bz2lib/1.0.8 libzstd/1.5.2
       envs directories : /opt/conda/envs
          package cache : /opt/conda/pkgs
                          /home/mambauser/.mamba/pkgs
            environment : base (active)
           env location : /opt/conda
      user config files : /home/mambauser/.mambarc
 populated config files : 
       virtual packages : __unix=0=0
                          __linux=5.15.49=0
                          __glibc=2.31=0
                          __archspec=1=x86_64
               channels : 
       base environment : /opt/conda
               platform : linux-64

Logs

$ docker run -it --rm -v $(pwd)/conda:/conda:ro  mambaorg/micromamba:latest micromamba env create -n my_env -f /conda/environment.yaml -y -v
info     libmamba Parsing MatchSpec pip=22.1.2
info     libmamba Parsing MatchSpec pip
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/linux-64/repodata.json'
info     libmamba No valid cache found
info     libmamba Using OpenSSL backend
info     libmamba Searching index cache file for repo 'https://conda.anaconda.org/conda-forge/noarch/repodata.json'
info     libmamba No valid cache found
info     libmamba All targets to download are cached
info     libmamba Starting to download targets
[+] 1.0s
conda-forge/linux-64 ━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   3.3MB /  29.5MB @   3.4MB/s  1.0s
conda-forge/noarch   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━  10.5MB /  11.9MB @  10.9MB/s  1.0sinfo     libmamba Transfer done for 'conda-forge/noarch'
conda-forge/noarch                                  11.9MB @  11.5MB/s  1.1s
[+] 3.0s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━  28.4MB /  29.5MB @   9.5MB/s  3.0sinfo     libmamba Transfer done for 'conda-forge/linux-64'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/repodata.jsoconda-forge/linux-64                                29.5MB @   9.7MB/s  3.1s
info     libmamba Reading cache files '/opt/conda/pkgs/cache/497deca9.json.*' for repo index 'conda-forge/linux-64'
info     libmamba Reading repodata.json file "/opt/conda/pkgs/cache/497deca9.json" for repo conda-forge/linux-64
info     libmamba Writing libsolv solv file "/opt/conda/pkgs/cache/497deca9.solv" for repo conda-forge/linux-64
info     libmamba Reading cache files '/opt/conda/pkgs/cache/09cdf8bf.json.*' for repo index 'conda-forge/noarch'
info     libmamba Reading repodata.json file "/opt/conda/pkgs/cache/09cdf8bf.json" for repo conda-forge/noarch
info     libmamba Writing libsolv solv file "/opt/conda/pkgs/cache/09cdf8bf.solv" for repo conda-forge/noarch
info     libmamba Adding package record to repo __archspec
info     libmamba Adding package record to repo __glibc
info     libmamba Adding package record to repo __linux
info     libmamba Adding package record to repo __unix
info     libmamba Parsing MatchSpec pip=22.1.2
info     libmamba Parsing MatchSpec pip=22.1.2
info     libmamba Adding job: pip=22.1.2
info     libmamba Parsing MatchSpec pip
info     libmamba Parsing MatchSpec pip
info     libmamba Adding job: pip
info     libmamba Problem count: 0
info     libmamba Found python version in packages to be installed 3.11.4

Transaction

  Prefix: /opt/conda/envs/my_env

  Updating specs:

   - pip=22.1.2
   - pip

  Package               Version  Build               Channel          Size
────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex           0.1  conda_forge         conda-forge       3kB
  + ld_impl_linux-64       2.40  h41732ed_0          conda-forge     705kB
  + ca-certificates   2023.7.22  hbcca054_0          conda-forge     150kB
  + libgomp              13.1.0  he5830b7_0          conda-forge     419kB
  + _openmp_mutex           4.5  2_gnu               conda-forge      24kB
  + libgcc-ng            13.1.0  he5830b7_0          conda-forge     776kB
  + openssl               3.1.2  hd590300_0          conda-forge       3MB
  + libzlib              1.2.13  hd590300_5          conda-forge      62kB
  + libffi                3.4.2  h7f98852_5          conda-forge      58kB
  + bzip2                 1.0.8  h7f98852_4          conda-forge     496kB
  + ncurses                 6.4  hcb278e6_0          conda-forge     881kB
  + libuuid              2.38.1  h0b41bf4_0          conda-forge      34kB
  + libexpat              2.5.0  hcb278e6_1          conda-forge      78kB
  + xz                    5.2.6  h166bdaf_0          conda-forge     418kB
  + libnsl                2.0.0  h7f98852_0          conda-forge      31kB
  + libsqlite            3.42.0  h2797004_0          conda-forge     829kB
  + tk                   8.6.12  h27826a3_0          conda-forge       3MB
  + readline                8.2  h8228510_1          conda-forge     281kB
  + tzdata                2023c  h71feb2d_0          conda-forge     118kB
  + python               3.11.4  hab00c5b_0_cpython  conda-forge      31MB
  + pip                  22.1.2  pyhd8ed1ab_0        conda-forge       2MB
  + setuptools           68.0.0  pyhd8ed1ab_0        conda-forge     464kB
  + wheel                0.41.0  pyhd8ed1ab_0        conda-forge      57kB

  Summary:

  Install: 23 packages

  Total download: 44MB

────────────────────────────────────────────────────────────────────────────

info     libmamba Cleaned 0 .mamba_trash files. 0 remaining.

Transaction starting
info     libmamba Starting to download targets
[+] 0.0s
Downloading  (1) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0.0 B /  44.3MB _libgcc_mutex              0.0s
Extracting       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━       0 /      23                            0.0sinfo     libmamba Transfer done for '_libgcc_mutex'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2] 2562 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/_libgcc_mutex-0.1-conda_forge.tar.bz2'
info     libmamba Transfer done for '_openmp_mutex'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/_openmp_mute_libgcc_mutex                                        2.6kB @  27.8kB/s  0.1s
_openmp_mutex                                       23.6kB @ 242.7kB/s  0.1s
[+] 0.1s
Downloading  (5) ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 390.6kB /  44.3MB ca-certificates            0.1s
Extracting   (2) ━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━       0 /      23 _libgcc_mutex              0.0sinfo     libmamba Transfer done for 'ca-certificates'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2023.7.22-hbcca054_0.conda] 149515 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/ca-certificates-2023.7.22-hbcca054_0.conda'
info     libmamba Transfer done for 'libgomp'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.1.0-he5830b7_0.conda] 419184 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/libgomp-13.1.0-he5830b7_0.conda'
info     libmamba Transfer done for 'ld_impl_linux-64'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linuca-certificates                                    149.5kB @   1.3MB/s  0.1s
libgomp                                            419.2kB @   3.5MB/s  0.1s
ld_impl_linux-64                                   704.7kB @   3.6MB/s  0.2s
[+] 0.2s
Downloading  (5) ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━   1.7MB /  44.3MB ncurses                    0.2s
Extracting   (4) ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━       0 /      23 _libgcc_mutex              0.1sinfo     libmamba Transfer done for 'wheel'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/noarch/wheel-0.41.0-pyhd8ed1ab_0.conda] 57356 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/wheel-0.41.0-pyhd8ed1ab_0.conda'
info     libmamba Transfer done for 'xz'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h16wheel                                               57.4kB @ 234.5kB/s  0.1s
xz                                                 418.4kB @   1.6MB/s  0.2s
[+] 0.3s
Downloading  (5) ━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━   3.5MB /  44.3MB ncurses                    0.3s
Extracting   (7) ━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━       0 /      23 _libgcc_mutex              0.2sinfo     libmamba Transfer done for 'libzlib'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.13-hd590300_5.conda] 61588 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/libzlib-1.2.13-hd590300_5.conda'
info     libmamba Transfer done for 'libuuid'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.38libzlib                                             61.6kB @ 179.6kB/s  0.1s
libuuid                                             33.6kB @  96.8kB/s  0.1s
[+] 0.5s
Downloading  (5) ━━━╸━━━━━━━━━━━━━━━━━━━━━━━╸━━━━   6.9MB /  44.3MB @  13.8MB/s ncurses                    0.5s
Extracting   (5) ━━━━╸━━━━━━━╸━━━━━━━━━━━━━━━━━━━       4 /      23             ld_impl_linux-64           0.4sinfo     libmamba Transfer done for 'ncurses'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.4-hcb278e6_0.conda] 880967 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/ncurses-6.4-hcb278e6_0.conda'
info     libmamba Transfer done for 'libsqlite'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libsqlite-3.42.0-h2797004_0.conda] 828910 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/libsqlite-3.42.0-h2797004_0.conda'
info     libmamba Transfer done for 'libexpat'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libexpat-2.5ncurses                                            881.0kB @   1.7MB/s  0.4s
libsqlite                                          828.9kB @   1.5MB/s  0.2s
libexpat                                            78.0kB @ 130.6kB/s  0.0s
[+] 0.6s
Downloading  (5) ━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━╸━━   9.0MB /  44.3MB @  13.8MB/s openssl                    0.6s
Extracting   (7) ━━━━╸━━━━━━━━━━╸━━━━━━━━━━━━━━━━       4 /      23             ld_impl_linux-64           0.5sinfo     libmamba Transfer done for 'tzdata'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/noarch/tzdata-2023c-htzdata                                             117.6kB @ 171.0kB/s  0.1s
[+] 0.7s
Downloading  (5) ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━╸━━  10.7MB /  44.3MB @  13.8MB/s openssl                    0.7s
Extracting   (6) ━━━━━━━╸━━━━━━━━╸━━━━━━━━━━━━━━━       6 /      23             ld_impl_linux-64           0.6sinfo     libmamba Transfer done for 'libffi'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2libffi                                              58.3kB @  75.1kB/s  0.1s
[+] 0.8s
Downloading  (5) ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━╸━━  12.8MB /  44.3MB @  13.8MB/s openssl                    0.8s
Extracting   (7) ━━━━━━━━╸━━━━━━━━━━╸━━━━━━━━━━━━       7 /      23             ld_impl_linux-64           0.7sinfo     libmamba Transfer done for 'readline'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/readline-8.2readline                                           281.5kB @ 316.9kB/s  0.1s
[+] 0.9s
Downloading  (5) ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━╸━  14.8MB /  44.3MB @  13.8MB/s openssl                    0.9s
Extracting   (6) ━━━━━━━━━━━╸━━━━━━━━╸━━━━━━━━━━━       9 /      23             libexpat                   0.8sinfo     libmamba Transfer done for 'pip'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/noarch/pip-22.1.2-pyhpip                                                  1.6MB @   1.7MB/s  0.6s
[+] 1.0s
Downloading  (5) ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━╸━  17.0MB /  44.3MB @  20.3MB/s python                     1.0s
Extracting   (6) ━━━━━━━━━━━━╸━━━━━━━━━╸━━━━━━━━━      10 /      23             ld_impl_linux-64           0.9sinfo     libmamba Transfer done for 'bzip2'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-bzip2                                              495.7kB @ 461.4kB/s  0.2s
[+] 1.2s
Downloading  (5) ━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━╸━  20.7MB /  44.3MB @  20.3MB/s python                     1.2s
Extracting   (6) ━━━━━━━━━━━━━━╸━━━━━━━━╸━━━━━━━━      11 /      23             ld_impl_linux-64           1.1sinfo     libmamba Transfer done for 'libnsl'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2] 31236 bytes
info     libmamba Download finished, validating '/opt/conda/pkgs/libnsl-2.0.0-h7f98852_0.tar.bz2'
info     libmamba Transfer done for 'tk'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h2libnsl                                              31.2kB @  25.6kB/s  0.1s
tk                                                   3.5MB @   2.8MB/s  1.1s
[+] 1.3s
Downloading  (4) ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━  22.5MB /  44.3MB @  20.3MB/s python                     1.3s
Extracting   (6) ━━━━━━━━━━━━━━━━━╸━━━━━━━━╸━━━━━      13 /      23             ld_impl_linux-64           1.2sinfo     libmamba Transfer done for 'libgcc-ng'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13libgcc-ng                                          776.3kB @ 581.2kB/s  0.4s
[+] 1.4s
Downloading  (3) ━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━  24.0MB /  44.3MB @  20.3MB/s setuptools                 1.4s
Extracting   (6) ━━━━━━━━━━━━━━━━━━╸━━━━━━━━╸━━━━      14 /      23             libgcc-ng                  1.3sinfo     libmamba Transfer done for 'setuptools'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/noarch/setuptools-68.setuptools                                         463.7kB @ 324.4kB/s  0.2s
[+] 1.6s
Downloading  (2) ━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━  27.5MB /  44.3MB @  17.5MB/s openssl                    1.6s
Extracting   (5) ━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━╸━━      16 /      23             libgcc-ng                  1.5sinfo     libmamba Transfer done for 'openssl'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/openssl-3.1.openssl                                              2.6MB @   1.6MB/s  1.2s
[+] 2.5s
Downloading  (1) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━  41.7MB /  44.3MB @  15.9MB/s python                     2.5s
Extracting       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━      22 /      23                                        2.4sinfo     libmamba Transfer done for 'python'
info     libmamba Transfer finalized, status: 200 [https://conda.anaconda.org/conda-forge/linux-64/python-3.11.python                                              30.7MB @  11.9MB/s  2.5s
Linking _libgcc_mutex-0.1-conda_forge
Linking ld_impl_linux-64-2.40-h41732ed_0
Linking ca-certificates-2023.7.22-hbcca054_0
Linking libgomp-13.1.0-he5830b7_0
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-ng-13.1.0-he5830b7_0
Linking openssl-3.1.2-hd590300_0
Linking libzlib-1.2.13-hd590300_5
Linking libffi-3.4.2-h7f98852_5
Linking bzip2-1.0.8-h7f98852_4
Linking ncurses-6.4-hcb278e6_0
Linking libuuid-2.38.1-h0b41bf4_0
Linking libexpat-2.5.0-hcb278e6_1
Linking xz-5.2.6-h166bdaf_0
Linking libnsl-2.0.0-h7f98852_0
Linking libsqlite-3.42.0-h2797004_0
Linking tk-8.6.12-h27826a3_0
Linking readline-8.2-h8228510_1
Linking tzdata-2023c-h71feb2d_0
Linking python-3.11.4-hab00c5b_0_cpython
Linking pip-22.1.2-pyhd8ed1ab_0
info     libmamba Running wrapped python compilation command /opt/conda/envs/my_env/bin/python3.11 -Wi -u /tmp/mambafNKow5S4xc6
info     libmamba Compiling 519 files to pyc
Linking setuptools-68.0.0-pyhd8ed1ab_0
info     libmamba Compiling 217 files to pyc
Linking wheel-0.41.0-pyhd8ed1ab_0
info     libmamba Compiling 27 files to pyc
info     libmamba Waiting for pyc compilation to finish

Transaction finished

To activate this environment, use:

    micromamba activate my_env

Or to execute a single command in this environment, use:

    micromamba run -n my_env mycommand

info     libmamba Opening history file: "/opt/conda/envs/my_env/conda-meta/history"
error    libmamba Error opening for writing "/conda/mambaffTnhAVCQfT": Read-only file system
error    libmamba Error opening for writing "/conda/mambaffTnhAVCQfT": Read-only file system

Installing pip packages: requests
info     libmamba Calling: /opt/conda/envs/my_env/bin/python -m pip install -r /conda/mambaffTnhAVCQfT --no-input
ERROR: Could not open requirements file: [Errno 2] No such file or directory: '/conda/mambaffTnhAVCQfT'
terminate called without an active exception

environment.yml

name: base

channels:
  - conda-forge

dependencies:
  - pip=22.1.2
  - pip:
    - requests

~/.condarc

No response

jonashaag commented 1 year ago

I think we have recently moved some other temp files directly to /tmp for similar reasons. Maybe it's time to replicate those changes everywhere in the code base