search

conda / conda-pack

Package conda environments for redistribution
https://conda.github.io/conda-pack/
BSD 3-Clause "New" or "Revised" License
519 stars 92 forks source link

permission error in conda-unpack on windows 10 #241

Open vickorian opened 1 year ago

vickorian commented 1 year ago

Checklist

What happened?

I'm moving an env in an attempt to create an offline system backup or system transfer. Issue is occurring when trying to transfer it to a test unit. Please see the attached screenshot. error in conda-unpack

Conda Info

active environment : base
    active env location : C:\Users\vicko\anaconda3
            shell level : 1
       user config file : C:\Users\vicko\.condarc
 populated config files : C:\Users\vicko\.condarc
          conda version : 22.9.0
    conda-build version : 3.22.0
         python version : 3.9.13.final.0
       virtual packages : __cuda=11.6=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\vicko\anaconda3  (writable)
      conda av data dir : C:\Users\vicko\anaconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\vicko\anaconda3\pkgs
                          C:\Users\vicko\.conda\pkgs
                          C:\Users\vicko\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\vicko\anaconda3\envs
                          C:\Users\vicko\.conda\envs
                          C:\Users\vicko\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.13 Windows/10 Windows/10.0.19044
          administrator : False
             netrc file : None
           offline mode : False

Conda Config

==> C:\Users\vicko\.condarc <==
channels:
  - defaults

Conda list

# packages in environment at C:\Users\vicko\anaconda3\envs\wt_:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
brotlipy                  0.7.0           py39h2bbff1b_1003    defaults
ca-certificates           2022.07.19           haa95532_0    defaults
certifi                   2022.9.24        py39haa95532_0    defaults
cffi                      1.15.1           py39h2bbff1b_0    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
colorama                  0.4.6                    pypi_0    pypi
cryptography              37.0.1           py39h21b164f_0    defaults
cudatoolkit               11.3.1               h59b6b97_2    defaults
ffmpeg                    1.4                      pypi_0    pypi
ffmpeg-python             0.2.0                    pypi_0    pypi
filelock                  3.8.0                    pypi_0    pypi
freetype                  2.10.4               hd328e21_0    defaults
future                    0.18.2           py39haa95532_1    defaults
huggingface-hub           0.10.1                   pypi_0    pypi
idna                      3.4              py39haa95532_0    defaults
intel-openmp              2021.4.0          haa95532_3556    defaults
jpeg                      9e                   h2bbff1b_0    defaults
lerc                      3.0                  hd77b12b_0    defaults
libdeflate                1.8                  h2bbff1b_5    defaults
libpng                    1.6.37               h2a8f88b_0    defaults
libtiff                   4.4.0                h8a3f274_0    defaults
libuv                     1.40.0               he774522_0    defaults
libwebp                   1.2.4                h2bbff1b_0    defaults
libwebp-base              1.2.4                h2bbff1b_0    defaults
lz4-c                     1.9.3                h2bbff1b_1    defaults
mkl                       2021.4.0           haa95532_640    defaults
mkl-service               2.4.0            py39h2bbff1b_0    defaults
mkl_fft                   1.3.1            py39h277e83a_0    defaults
mkl_random                1.2.2            py39hf11a4ad_0    defaults
more-itertools            9.0.0                    pypi_0    pypi
ninja                     1.10.2               haa95532_5    defaults
ninja-base                1.10.2               h6d14046_5    defaults
numpy                     1.23.3           py39h3b20f71_0    defaults
numpy-base                1.23.3           py39h4da318b_0    defaults
openssl                   1.1.1q               h2bbff1b_0    defaults
packaging                 21.3                     pypi_0    pypi
pillow                    9.2.0            py39hdc2b20a_1    defaults
pip                       22.2.2           py39haa95532_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 22.0.0             pyhd3eb1b0_0    defaults
pyparsing                 3.0.9                    pypi_0    pypi
pysocks                   1.7.1            py39haa95532_0    defaults
python                    3.9.13               h6244533_2    defaults
pytorch                   1.12.1          py3.9_cuda11.3_cudnn8_0    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pyyaml                    6.0                      pypi_0    pypi
regex                     2022.9.13                pypi_0    pypi
requests                  2.28.1           py39haa95532_0    defaults
setuptools                63.4.1           py39haa95532_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.39.3               h2bbff1b_0    defaults
tk                        8.6.12               h2bbff1b_0    defaults
tokenizers                0.13.1                   pypi_0    pypi
torchaudio                0.12.1               py39_cu113    pytorch
torchvision               0.13.1               py39_cu113    pytorch
tqdm                      4.64.1                   pypi_0    pypi
transformers              4.23.1                   pypi_0    pypi
typing-extensions         4.3.0            py39haa95532_0    defaults
typing_extensions         4.3.0            py39haa95532_0    defaults
tzdata                    2022e                h04d1e81_0    defaults
urllib3                   1.26.12          py39haa95532_0    defaults
vc                        14.2                 h21ff451_1    defaults
vs2015_runtime            14.27.29016          h5e58377_2    defaults
wheel                     0.37.1             pyhd3eb1b0_0    defaults
whisper                   1.0                      pypi_0    pypi
win_inet_pton             1.1.0            py39haa95532_0    defaults
wincertstore              0.2              py39haa95532_2    defaults
xz                        5.2.6                h8cc25b3_0    defaults
zlib                      1.2.13               h8cc25b3_0    defaults
zstd                      1.5.2                h19a0ad4_0    defaults

Additional Context

No response

vickorian commented 1 year ago

After some testing it's one of the following packages causing the error.

colorama=0.4.6=pypi_0 ffmpeg-python=0.2.0=pypi_0 filelock=3.8.0=pypi_0 future=0.18.2=pypi_0 huggingface-hub=0.10.1=pypi_0 idna=3.4=py39haa95532_0 more-itertools=9.0.0=pypi_0 packaging=21.3=pypi_0 pyparsing=3.0.9=pypi_0 pyyaml=6.0=pypi_0 regex=2022.9.13=pypi_0 tokenizers=0.13.1=pypi_0 tqdm=4.64.1=pypi_0 transformers=4.23.1=pypi_0

vickorian commented 1 year ago

ffmpeg-python=0.2.0=pypi_0 more-itertools=9.0.0=pypi_0 pyyaml=6.0=pypi_0
regex=2022.9.13=pypi_0

There is a conflict in one of these four.

axel-kah commented 1 year ago

I can confirm that these PermissionErrors still exist in the latest conda version 23.1.0 running on windows 10 22H2. In my case it failed to patch the prefix of a .pyd file of wrapt in the site packages folder of the environment I tried to unpack. The issue is also reproducible (with the same file being affected) on dedicated windows CI nodes that do not use any "file monitoring" solutions, which is usually a good candidate for these kinds of problems.

What @vickorian and I have in common is that we try to use conda-unpack.exe. So I tried the documented alternative conda-unpack-script.py with the interpeter that is part of the env I'm trying to unpack. Same problem. Then I used the script but with an independent python interpreter - and it worked reliably.

Hypothesis

The conda-unpack.exe uses the interpreter that is part of the environment to be patched. In other words, the environment is used to patch itself. Maybe that is just a bad idea in general and the robust way is to always use an independent interpreter to perform this job.

conda info

$ conda info

     active environment : base
    active env location : C:\workflow2\msys2\opt\workflow\workflow_conda_scientific_env_1.6.0.0
            shell level : 1
       user config file : C:\Users\john\.condarc
 populated config files : C:\workflow2\msys2\opt\workflow\workflow_conda_scientific_env_1.6.0.0\.condarc
                          C:\Users\john\.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.9.16.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=11.7=0
                          __win=0=0
       base environment : C:\workflow2\msys2\opt\workflow\workflow_conda_scientific_env_1.6.0.0  (writable)
      conda av data dir : C:\workflow2\msys2\opt\workflow\workflow_conda_scientific_env_1.6.0.0\etc\conda
  conda av metadata url : None
           channel URLs : https://artifactory.intra.somecorp.com/artifactory/api/conda/conda-anaconda-pkgs-main-virtual/win-64
                          https://artifactory.intra.somecorp.com/artifactory/api/conda/conda-anaconda-pkgs-main-virtual/noarch
                          https://artifactory.intra.somecorp.com/artifactory/api/conda/conda-anaconda-pkgs-msys2-virtual/win-64
                          https://artifactory.intra.somecorp.com/artifactory/api/conda/conda-anaconda-pkgs-msys2-virtual/noarch
                          https://artifactory.intra.somecorp.com/artifactory/api/conda/conda-anaconda-pkgs-r-virtual/win-64
                          https://artifactory.intra.somecorp.com/artifactory/api/conda/conda-anaconda-pkgs-r-virtual/noarch
          package cache : C:\workflow2\msys2\opt\workflow\workflow_conda_scientific_env_1.6.0.0\pkgs
       envs directories : C:\Users\john\AppData\Local\conda\conda\envs
                          C:\workflow2\msys2\opt\workflow
                          C:\workflow2\msys2\opt\workflow\workflow_conda_scientific_env_1.6.0.0\envs
                          C:\Users\john\.conda\envs
               platform : win-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.9.16 Windows/10 Windows/10.0.19045
          administrator : False
             netrc file : None
           offline mode : False
ZhiZe-ZG commented 11 months ago

@axel-kah your solution works. Use the interpreter of base environment to run the <env name>/Script/conda-unpack-script.py script. Thanks.

btsai-dev commented 8 months ago

I also second @ZhiZe-ZG's comment.

Specifically, running python <env name>/Scripts/conda-unpack-script.py as described in the directory above the environment name. Running python Scripts/conda-unpack-script.py from within the environment folder still resulted in permission errors.