search

conda / conda-build

Commands and tools for building conda packages
https://docs.conda.io/projects/conda-build/
Other
383 stars 423 forks source link

merge_tree: bug in checking for subdirectory #4976

Open woutdenolf opened 1 year ago

woutdenolf commented 1 year ago

Checklist

What happened?

$ call conda mambabuild recipe --prefix-length=80 --output-folder=conda_packages -c esrf-bcu -c conda-forge

No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.22
Updating build index: C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss\conda_packages
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for bliss
INFO:conda_build.metadata:Attempting to finalize metadata for bliss
Attempting to finalize metadata for bliss
INFO:conda_build.metadata:Attempting to finalize metadata for bliss
Reloading output folder: 
C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss\conda_packages
Reloading output folder: 
C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss\conda_packages
Reloading output folder: 
C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss\conda_packages
BUILD START: ['bliss-1.12.dev0-pyh74a9793_130116.tar.bz2']
Reloading output folder: 
C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss\conda_packages
Reloading output folder: 
C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss\conda_packages
## Package Plan ##
  environment location: C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\conda-bld\bliss_1692732999059\_h_env
The following NEW packages will be INSTALLED:
    bzip2:           1.0.8-h8ffe710_4          conda-forge
    ca-certificates: 2023.7.22-h56e8100_0      conda-forge
    libexpat:        2.5.0-h63175ca_1          conda-forge
    libffi:          3.4.2-h8ffe710_5          conda-forge
    libsqlite:       3.42.0-hcfcfb64_0         conda-forge
    libzlib:         1.2.13-hcfcfb64_5         conda-forge
    openssl:         3.1.2-hcfcfb64_0          conda-forge
    pip:             23.2.1-pyhd8ed1ab_0       conda-forge
    python:          3.11.4-h2628c8c_0_cpython conda-forge
    setuptools:      68.1.2-pyhd8ed1ab_0       conda-forge
    tk:              8.6.12-h8ffe710_0         conda-forge
    tzdata:          2023c-h71feb2d_0          conda-forge
    ucrt:            10.0.22621.0-h57928b3_0   conda-forge
    vc:              14.3-h64f974e_17          conda-forge
    vc14_runtime:    14.36.32532-hfdfe4a8_17   conda-forge
    vs2015_runtime:  14.36.32532-h05e6639_17   conda-forge
    wheel:           0.41.2-pyhd8ed1ab_0       conda-forge
    xz:              5.2.6-h8d14728_0          conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Copying C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss to C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\conda-bld\bliss_1692732999059\work\
Traceback (most recent call last):
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Scripts\conda-mambabuild-script.py", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\boa\cli\mambabuild.py", line 256, in main
    call_conda_build(action, config)
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\boa\cli\mambabuild.py", line 228, in call_conda_build
    result = api.build(
             ^^^^^^^^^^
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\api.py", line 253, in build
    return build_tree(
           ^^^^^^^^^^^
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\build.py", line 3799, in build_tree
    packages_from_this = build(
                         ^^^^^^
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\build.py", line 2582, in build
    try_download(m, no_download_source=False, raise_error=True)
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\render.py", line 757, in try_download
    source.provide(metadata)
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\source.py", line 1100, in provide
    copy_into(
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\utils.py", line 616, in copy_into
    merge_tree(
  File "C:\GitLab-Runner\builds\Dx9KqxMZ\0\bliss\bliss-build-env\Lib\site-packages\conda_build\utils.py", line 729, in merge_tree
    assert not dst.startswith(src), (
AssertionError: Can't merge/copy source into subdirectory of itself.  Please create separate spaces for these things.
  src: c:\gitlab-runner\builds\dx9kqxmz\0\bliss\bliss
  dst: c:\gitlab-runner\builds\dx9kqxmz\0\bliss\bliss-build-env\conda-bld\bliss_1692732999059\work

Conda Info

active environment : base
    active env location : C:\ProgramData\miniconda3
            shell level : 1
       user config file : C:\WINDOWS\system32\config\systemprofile\.condarc
 populated config files : C:\Windows\System32\config\systemprofile\.condarc
          conda version : 23.7.2
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __win=0=0
       base environment : C:\ProgramData\miniconda3  (writable)
      conda av data dir : C:\ProgramData\miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\ProgramData\miniconda3\pkgs
                          C:\WINDOWS\system32\config\systemprofile\.conda\pkgs
                          C:\WINDOWS\system32\config\systemprofile\AppData\Local\conda\conda\pkgs
       envs directories : C:\ProgramData\miniconda3\envs
                          C:\WINDOWS\system32\config\systemprofile\.conda\envs
                          C:\WINDOWS\system32\config\systemprofile\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.7.2 requests/2.29.0 CPython/3.10.12 Windows/10 Windows/10.0.19044
          administrator : True
             netrc file : None
           offline mode : False

Conda Config

==> cmd_line <==
debug: False
json: False

Conda list

# packages in environment at C:\ProgramData\miniconda3:
#
# Name                    Version                   Build  Channel
boltons                   23.0.0          py310haa95532_0    defaults
brotlipy                  0.7.0           py310h2bbff1b_1002    defaults
bzip2                     1.0.8                he774522_0    defaults
ca-certificates           2023.7.22            h56e8100_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h2bbff1b_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
colorama                  0.4.6           py310haa95532_0    defaults
conda                     23.7.2          py310h5588dad_0    conda-forge
conda-content-trust       0.1.3           py310haa95532_0    defaults
conda-libmamba-solver     23.5.0          py310haa95532_0    defaults
conda-package-handling    2.1.0           py310haa95532_0    defaults
conda-package-streaming   0.8.0           py310haa95532_0    defaults
console_shortcut_miniconda 0.1.1                haa95532_1    defaults
cryptography              39.0.1          py310h21b164f_2    defaults
fmt                       9.1.0                h6d14046_0    defaults
idna                      3.4             py310haa95532_0    defaults
jsonpatch                 1.32               pyhd3eb1b0_0    defaults
jsonpointer               2.1                pyhd3eb1b0_0    defaults
krb5                      1.21.2               heb0366b_0    conda-forge
libarchive                3.6.2                hb62f4d4_2    defaults
libcurl                   8.2.1                hd5e4a3a_0    conda-forge
libffi                    3.4.4                hd77b12b_0    defaults
libiconv                  1.16                 h2bbff1b_2    defaults
libmamba                  1.4.9                ha24f096_0    conda-forge
libmambapy                1.4.9           py310h3fe4c2e_0    conda-forge
libsolv                   0.7.24               h12be248_3    conda-forge
libssh2                   1.11.0               h7dfc565_0    conda-forge
libxml2                   2.10.3               h0ad7f3c_0    defaults
libzlib                   1.2.13               hcfcfb64_5    conda-forge
lz4-c                     1.9.4                h2bbff1b_0    defaults
mamba                     1.4.9           py310hd9d798f_0    conda-forge
menuinst                  1.4.19          py310h59b6b97_0    defaults
openssl                   3.1.2                hcfcfb64_0    conda-forge
packaging                 23.0            py310haa95532_0    defaults
pcre2                     10.37                h0ff8eda_1    defaults
pip                       23.1.2          py310haa95532_0    defaults
pluggy                    1.0.0           py310haa95532_1    defaults
powershell_shortcut_miniconda 0.0.1                haa95532_1    defaults
pybind11-abi              4                    hd3eb1b0_1    defaults
pycosat                   0.6.4           py310h2bbff1b_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pyopenssl                 23.0.0          py310haa95532_0    defaults
pysocks                   1.7.1           py310haa95532_0    defaults
python                    3.10.12              he1021f5_0    defaults
python_abi                3.10                    2_cp310    conda-forge
reproc                    14.2.4               hd77b12b_1    defaults
reproc-cpp                14.2.4               hd77b12b_1    defaults
requests                  2.29.0          py310haa95532_0    defaults
ruamel.yaml               0.17.21         py310h2bbff1b_0    defaults
ruamel.yaml.clib          0.2.6           py310h2bbff1b_1    defaults
setuptools                67.8.0          py310haa95532_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.41.2               h2bbff1b_0    defaults
tk                        8.6.12               h2bbff1b_0    defaults
toolz                     0.12.0          py310haa95532_0    defaults
tqdm                      4.65.0          py310h9909e9c_0    defaults
tzdata                    2023c                h04d1e81_0    defaults
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   1.26.16         py310haa95532_0    defaults
vc                        14.2                 h21ff451_1    defaults
vc14_runtime              14.36.32532         hfdfe4a8_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.38.4          py310haa95532_0    defaults
win_inet_pton             1.1.0           py310haa95532_0    defaults
xz                        5.4.2                h8cc25b3_0    defaults
yaml-cpp                  0.7.0                hd77b12b_1    defaults
zlib                      1.2.13               hcfcfb64_5    conda-forge
zstandard                 0.19.0          py310h2bbff1b_0    defaults
zstd                      1.5.5                hd43e919_0    defaults

Additional Context

No response

julian-belina commented 10 months ago

I have a similar problem when trying to build a conda package on a Gitlab CI runner. The process still works on a local windows machine though CURRENT_DIRECTORY = Path(__file__).parent.absolute() OUTPUT_FOLDER = os.path.join(CURRENT_DIRECTORY, r"local_conda_packages_temp") subprocess.run( [ "conda", "mambabuild", os.path.join(CURRENT_DIRECTORY, "meta_yamls", "fine"), "--output-folder", OUTPUT_FOLDER, "-c", "conda-forge", "-r", os.path.join(CURRENT_DIRECTORY, "conda_package_folder"), ], check=True, )

WARNING: No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.22 Updating build index: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp Adding in variants from internal_defaults Attempting to finalize metadata for fine conda-forge/linux-64 Using cache conda-forge/noarch Using cache Reloading output folder: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp Reloading output folder: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp Reloading output folder: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp BUILD START: ['fine-2.2.3-pypy_0.tar.bz2'] Reloading output folder: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp Package Plan environment location: /opt/conda/conda-bld/fine_1705083043911/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol The following NEW packages will be INSTALLED: _libgcc_mutex: 0.1-conda_forge conda-forge _openmp_mutex: 4.5-2_gnu conda-forge bzip2: 1.0.8-hd590300_5 conda-forge ca-certificates: 2023.11.17-hbcca054_0 conda-forge ld_impl_linux-64: 2.40-h41732ed_0 conda-forge libexpat: 2.5.0-hcb278e6_1 conda-forge libffi: 3.4.2-h7f98852_5 conda-forge libgcc-ng: 13.2.0-h807b86a_3 conda-forge libgomp: 13.2.0-h807b86a_3 conda-forge libnsl: 2.0.1-hd590300_0 conda-forge libsqlite: 3.44.2-h2797004_0 conda-forge libuuid: 2.38.1-h0b41bf4_0 conda-forge libxcrypt: 4.4.36-hd590300_1 conda-forge libzlib: 1.2.13-hd590300_5 conda-forge ncurses: 6.4-h59595ed_2 conda-forge openssl: 3.2.0-hd590300_1 conda-forge pip: 23.3.2-pyhd8ed1ab_0 conda-forge python: 3.12.1-hab00c5b_1_cpython conda-forge readline: 8.2-h8228510_1 conda-forge setuptools: 69.0.3-pyhd8ed1ab_0 conda-forge setuptools-git: 1.2-py_1 conda-forge tk: 8.6.13-noxft_h4845f30_101 conda-forge tzdata: 2023d-h0c530f3_0 conda-forge wheel: 0.42.0-pyhd8ed1ab_0 conda-forge xz: 5.2.6-h166bdaf_0 conda-forge Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done Reloading output folder: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp Reloading output folder: /builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp Conda package build process starts Traceback (most recent call last): File "/builds/iek-3/shared-code/ethos-installation/create_conda_packages.py", line 8, in <module> subprocess.run( File "/opt/conda/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['conda', 'mambabuild', '/builds/iek-3/shared-code/ethos-installation/meta_yamls/fine', '--output-folder', '/builds/iek-3/shared-code/ethos-installation/local_conda_packages_temp', '-c', 'conda-forge', '-r', '/builds/iek-3/shared-code/ethos-installation/conda_package_folder']' died with <Signals.SIGKILL: 9>.

I run the command from this environment: ` Install:

nikitakuklev commented 8 months ago

Still a relevant issue. A workaround is to set CONDA_BLD_PATH to ensure no overlap.