Conda-Managed Files Deleted or Overwritten

GitPeterJ commented 10 months ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

While trying to create a packaged environment for Triton server I get an error when I run the conda-pack command. In the error I see it's trying to get packages from the python3.1 folder, I saw a previous bug report where this was fixed but apparently, it's still a problem.

*ERROR

Collecting packages... CondaPackError: Files managed by conda were found to have been deleted/overwritten in the following packages:

pycparser 2.21: lib/python3.1/site-packages/pycparser-2.21.dist-info/INSTALLER lib/python3.1/site-packages/pycparser-2.21.dist-info/LICENSE lib/python3.1/site-packages/pycparser-2.21.dist-info/METADATA
- 23 others

This is usually due to pip uninstalling or clobbering conda managed files, resulting in an inconsistent environment. Please check your environment for conda/pip conflicts using conda list, and fix the environment by ensuring only one version of each package is installed (conda preferred).

Conda Info

Env: Ubuntu wsl

Conda Config

channels:
  - defaults
  - conda-forge
  - pytorch
  - nvidia
dependencies:
  - python=3.10
  - pytorch
  - pytorch-cuda=12.1
  - pip
  - pip:
    - transformers[accelerate,sentencepiece]

Conda list

_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
accelerate                0.23.0                   pypi_0    pypi
blas                      1.0                         mkl    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
ca-certificates           2023.08.22           h06a4308_0    defaults
certifi                   2023.7.22                pypi_0    pypi
cffi                      1.15.1          py311h5eee18b_3    defaults
charset-normalizer        3.3.0                    pypi_0    pypi
cuda-cudart               12.1.105                      0    nvidia
cuda-cupti                12.1.105                      0    nvidia
cuda-libraries            12.1.0                        0    nvidia
cuda-nvrtc                12.1.105                      0    nvidia
cuda-nvtx                 12.1.105                      0    nvidia
cuda-opencl               12.3.52                       0    nvidia
cuda-runtime              12.1.0                        0    nvidia
filelock                  3.9.0           py311h06a4308_0    defaults
fsspec                    2023.9.2                 pypi_0    pypi
gmp                       6.2.1                h295c915_3    defaults
gmpy2                     2.1.2           py311hc9b5ff0_0    defaults
huggingface-hub           0.17.3                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
intel-openmp              2023.1.0         hdb19cb5_46305    defaults
jinja2                    3.1.2           py311h06a4308_0    defaults
ld_impl_linux-64          2.38                 h1181459_1    defaults
libcublas                 12.1.0.26                     0    nvidia
libcufft                  11.0.2.4                      0    nvidia
libcufile                 1.8.0.34                      0    nvidia
libcurand                 10.3.4.52                     0    nvidia
libcusolver               11.4.4.55                     0    nvidia
libcusparse               12.0.2.55                     0    nvidia
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libnpp                    12.0.2.50                     0    nvidia
libnvjitlink              12.1.105                      0    nvidia
libnvjpeg                 12.1.1.14                     0    nvidia
libprotobuf               3.20.3               he621ea3_0    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
markupsafe                2.1.1           py311h5eee18b_0    defaults
mkl                       2023.1.0         h213fc3f_46343    defaults
mkl-service               2.4.0           py311h5eee18b_1    defaults
mkl_fft                   1.3.8           py311h5eee18b_0    defaults
mkl_random                1.2.4           py311hdb19cb5_0    defaults
mpc                       1.1.0                h10f8cd9_1    defaults
mpfr                      4.0.2                hb69a4c5_1    defaults
mpmath                    1.3.0           py311h06a4308_0    defaults
ncurses                   6.4                  h6a678d5_0    defaults
networkx                  3.1             py311h06a4308_0    defaults
ninja                     1.10.2               h06a4308_5    defaults
ninja-base                1.10.2               hd09550d_5    defaults
numpy                     1.26.0          py311h08b1b3b_0    defaults
numpy-base                1.26.0          py311hf175353_0    defaults
openssl                   3.0.11               h7f8727e_2    defaults
packaging                 23.2                     pypi_0    pypi
pip                       23.3            py311h06a4308_0    defaults
protobuf                  4.24.4                   pypi_0    pypi
psutil                    5.9.6                    pypi_0    pypi
pycparser                 2.21               pyhd3eb1b0_0    defaults
python                    3.11.5               h955ad1f_0    defaults
pytorch                   2.0.1           cpu_py311h6d93b4c_0    defaults
pytorch-cuda              12.1                 ha16c6d3_5    pytorch
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0    defaults
regex                     2023.10.3                pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
safetensors               0.4.0                    pypi_0    pypi
sentencepiece             0.1.99                   pypi_0    pypi
setuptools                68.0.0          py311h06a4308_0    defaults
sqlite                    3.41.2               h5eee18b_0    defaults
sympy                     1.11.1          py311h06a4308_0    defaults
tbb                       2021.8.0             hdb19cb5_0    defaults
tk                        8.6.12               h1ccaba5_0    defaults
tokenizers                0.14.1                   pypi_0    pypi
tqdm                      4.66.1                   pypi_0    pypi
transformers              4.34.1                   pypi_0    pypi
typing-extensions         4.7.1           py311h06a4308_0    defaults
typing_extensions         4.7.1           py311h06a4308_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
urllib3                   2.0.7                    pypi_0    pypi
wheel                     0.41.2          py311h06a4308_0    defaults
xz                        5.4.2                h5eee18b_0    defaults
zlib                      1.2.13               h5eee18b_0    defaults

Additional Context

No response

albertma1986 commented 9 months ago

I had the same thing, I was using python 3.10. My gut feeling is the process involves snipping the "3.??" string hence it was trying to refer the "python3.1/...." paths.

I switched to python 3.9 and this error disappeared. But you are right this is indeed a bug :)

GitPeterJ commented 9 months ago

The problem is that the version is being snipped, while searching for a solution I saw that the first approach was to do it like str[:3], but this was already fixed (don't remember which commit). So I am confused about why this is still a problem, 2 years after 3.10 was released! Also downgrading to 3.9 should not be a final solution as this version will be phased out.

Fallqs commented 4 days ago

An unsynchronized conda-meta directory can cause the same problem. When updating certain packages, conda can update the packages without updating corresonding json files in conda-meta (i.e. updating pytorch 2.4.0-cuda12.1 from 2.1.2-cuda12.1). Such behavior does not affect the functions of conda environment, but leads to confusion while launching conda pack. I think a force mode that ignores conda-meta and simply pack any files within the env directory can be a solution.

conda / conda-pack