search

bioconda / bioconda-utils

Utilities for building and managing bioconda recipes
MIT License
96 stars 133 forks source link

`mulled-build` of Python package (no deps) fails in bioconda-utils >=2.15 #990

Closed gregdenay closed 3 months ago

gregdenay commented 3 months ago

I have a Python package with minimal build dependencies and no runtime dependencies currently being submitted as a new recipe in bioconda: PR here.

The bioconda-utils checks fail on the mulled-buildstep with a an unsatisfiable error for __glibc:

14:22:22 BIOCONDA INFO (COMMAND) mulled-build build-and-test taxidtools=2.3.0--pyhdfd78af_0 -n biocontainers --test bash -c '/usr/local/env-execute true && . /usr/local/env-activate.sh && pip check' --channels conda-forge,file:///home/debian/anaconda3/envs/bioconda/conda-bld,bioconda,defaults --involucro-path /home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/bioconda_utils/involucro
14:22:23 BIOCONDA INFO (OUT) Executing: /home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/bioconda_utils/involucro -v=2 -f /home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,file:///home/debian/anaconda3/envs/bioconda/conda-bld,bioconda,defaults -set TARGETS=taxidtools=2.3.0=pyhdfd78af_0 -set REPO=quay.io/biocontainers/taxidtools:2.3.0--pyhdfd78af_0 -set BINDS=build/dist:/usr/local/,/home/debian/anaconda3/envs/bioconda/conda-bld:/home/debian/anaconda3/envs/bioconda/conda-bld -set DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:3.0 -set CONDA_IMAGE=quay.io/bioconda/create-env:latest -set 'TEST=bash -c '"'"'/usr/local/env-execute true && . /usr/local/env-activate.sh && pip check'"'"'' -set CONDA_BIN=conda build-and-test
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Run file [/home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] INFO Invoke Task [build]
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] STEP Run image [quay.io/bioconda/create-env:latest] with command [[rm -rf /data/dist]]
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Creating container [step-860fbe1f0c]
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Created container [5baec6a11350 step-860fbe1f0c], starting it
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Container [5baec6a11350 step-860fbe1f0c] started, waiting for completion
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Container [5baec6a11350 step-860fbe1f0c] completed with exit code [0] as expected
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Container [5baec6a11350 step-860fbe1f0c] removed
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] STEP Run image [quay.io/bioconda/create-env:latest] with command [[/bin/sh -c conda install  -c 'conda-forge' -c 'file:///home/debian/anaconda3/envs/bioconda/conda-bld' -c 'bioconda' -c 'defaults'  'taxidtools=2.3.0=pyhdfd78af_0' --strict-channel-priority -p /usr/local --copy --yes --quiet&&create-env --conda=: /usr/local]]
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Creating container [step-ffe0855f42]
14:22:23 BIOCONDA INFO (ERR) [May 13 14:22:23] DEBU Created container [297a365c3bac step-ffe0855f42], starting it
14:22:24 BIOCONDA INFO (ERR) [May 13 14:22:24] DEBU Container [297a365c3bac step-ffe0855f42] started, waiting for completion
..14:24:39 BIOCONDA INFO (ERR) [May 13 14:24:39] SOUT Collecting package metadata (repodata.json): ...working... done
14:24:44 BIOCONDA INFO (ERR) [May 13 14:24:44] SOUT Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
14:24:53 BIOCONDA INFO (ERR) [May 13 14:24:53] SOUT Solving environment: ...working...
14:24:53 BIOCONDA INFO (ERR) [May 13 14:24:53] SOUT Found conflicts! Looking for incompatible packages.
14:24:53 BIOCONDA INFO (ERR) [May 13 14:24:53] SOUT This can take several minutes.  Press CTRL-C to abort.
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SOUT failed
Examining conflict for __glibc python:   0%|          | 0/5 [00:00<?, ?it/s]               | 0/5 [00:00<?, ?it/s]
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR UnsatisfiableError: The following specifications were found to be incompatible with your system:
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR - feature:/linux-64::__glibc==2.36=0
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR - python[version='>=3.9'] -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR Your installed version is: 2.36
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR Note that strict channel priority may have removed packages required for satisfiability.
14:24:55 BIOCONDA INFO (ERR) [May 13 14:24:55] SERR

It seems weird that python requiring __glibc >= 2.17 cannot be satisfied with __glibc==2.36 and this build worked fine with bioconda-utils 2.14, so I guess this is the correct place to post the issue.

The complete logs are availble from the PR. You should be able to reproduce it locally with:

git clone https://github.com/gregdenay/bioconda-recipes.git
cd bioconda-recipes
git checkout origin/add_taxidTools
bioconda-utils build --docker --mulled-test --packages taxidtools

If this is not a problem with bioconda-utils itself, I would still love some hepp in finding out what the problem is :)

aliciaaevans commented 3 months ago

I'm not entirely sure what's going on here yet, but upcoming changes to bioconda-utils requirements and pinnings may resolve it.

gregdenay commented 3 months ago

Version 3 did not solve it. It appears to be a similar error but the logging is less informative than in the previous version:

08:21:35 BIOCONDA INFO (ERR) [May 31 08:21:35] STEP Run image [quay.io/bioconda/create-env:latest                                                      ] with command [[/bin/sh -c conda install  -c 'conda-forge' -c 'file:///home/debian/anaconda3/env                                                      s/bioconda/conda-bld' -c 'bioconda'  'taxidtools=2.3.0=pyhdfd78af_0' --strict-channel-priority -p                                                       /usr/local --copy --yes --quiet&&create-env --conda=: /usr/local]]
08:21:35 BIOCONDA INFO (ERR) [May 31 08:21:35] DEBU Creating container [step-f0c0ce069d]
08:21:36 BIOCONDA INFO (ERR) [May 31 08:21:36] DEBU Created container [7bf94c21b576 step-f0c0ce06                                                      9d], starting it
08:21:36 BIOCONDA INFO (ERR) [May 31 08:21:36] DEBU Container [7bf94c21b576 step-f0c0ce069d] star                                                      ted, waiting for completion
..08:23:42 BIOCONDA INFO (ERR) [May 31 08:23:42] SOUT Collecting package metadata (repodata.json)                                                      : ...working... done
08:23:45 BIOCONDA INFO (ERR) [May 31 08:23:45] SOUT Solving environment: ...working... failed wit                                                      h initial frozen solve. Retrying with flexible solve.
08:23:53 BIOCONDA INFO (ERR) [May 31 08:23:53] SOUT Solving environment: ...working...
08:23:53 BIOCONDA INFO (ERR) [May 31 08:23:53] SOUT Found conflicts! Looking for incompatible pac                                                      kages.
08:23:53 BIOCONDA INFO (ERR) [May 31 08:23:53] SOUT This can take several minutes.  Press CTRL-C                                                       to abort.
08:23:54 BIOCONDA INFO (ERR) [May 31 08:23:54] SOUT failed
08:23:54 BIOCONDA INFO (ERR) [May 31 08:23:54] SERR Building graph of deps:   0%|          | 0/5                                                       Determining conflicts:   0%|          | 0/5 [00:00<?, ?it/s]
08:23:54 BIOCONDA INFO (ERR) [May 31 08:23:54] SERR UnsatisfiableError:
08:23:54 BIOCONDA INFO (ERR) [May 31 08:23:54] SERR Note that strict channel priority may have re                                                      moved packages required for satisfiability.
08:23:54 BIOCONDA INFO (ERR) [May 31 08:23:54] SERR
08:24:02 BIOCONDA INFO (ERR) [May 31 08:24:02] ERRO Task processing failed: Unexpected exit code                                                       [1] of container [7bf94c21b576 step-f0c0ce069d], container preserved
08:24:02 BIOCONDA INFO (OUT) ...
08:24:02 BIOCONDA ERROR COMMAND FAILED (exited with 1): mulled-build build-and-test taxidtools=2.                                                      3.0--pyhdfd78af_0 -n biocontainers --test bash -c '/usr/local/env-execute true && . /usr/local/en                                                      v-activate.sh && pip check' --channels conda-forge,file:///home/debian/anaconda3/envs/bioconda/co                                                      nda-bld,bioconda --involucro-path /home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packag                                                      es/bioconda_utils/involucro

08:24:02 BIOCONDA ERROR TEST FAILED: recipes/taxidtools
08:24:02 BIOCONDA INFO (COMMAND) conda build purge
08:24:03 BIOCONDA ERROR BUILD SUMMARY: of 1 recipes, 1 failed and 0 were skipped. Details of reci                                                      pes and environments follow.
08:24:03 BIOCONDA ERROR BUILD SUMMARY: FAILED recipe recipes/taxidtools

Do you know where/if I can find the full logs? I tried docker images | head as per the bioconda doc to find the container but nothing recent shows up.

gregdenay commented 3 months ago

It seems that the last changes in bioconda-utils fixed the gcc problems:

08:26:36 BIOCONDA INFO (OUT) Executing: /home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/bioconda_utils/involucro -v=2 -f /home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua -set CHANNELS=conda-forge,file:///home/debian/anaconda3/envs/bioconda/conda-bld,bioconda -set TARGETS=taxidtools=2.3.0=pyhdfd78af_0 -set REPO=quay.io/biocontainers/taxidtools:2.3.0--pyhdfd78af_0 -set BINDS=build/dist:/usr/local/,/home/debian/anaconda3/envs/bioconda/conda-bld:/home/debian/anaconda3/envs/bioconda/conda-bld -set DEST_BASE_IMAGE=quay.io/bioconda/base-glibc-busybox-bash:3.1 -set CONDA_IMAGE=quay.io/bioconda/create-env:latest -set 'TEST=bash -c '"'"'/usr/local/env-execute true && . /usr/local/env-activate.sh && pip check'"'"'' -set CONDA_BIN=conda build-and-test
08:26:36 BIOCONDA INFO (ERR) [Jun  4 08:26:36] DEBU Run file [/home/debian/anaconda3/envs/bioconda/lib/python3.10/site-packages/galaxy/tool_util/deps/mulled/invfile.lua]
08:26:36 BIOCONDA INFO (ERR) [Jun  4 08:26:36] INFO Invoke Task [build]
08:26:36 BIOCONDA INFO (ERR) [Jun  4 08:26:36] STEP Run image [quay.io/bioconda/create-env:latest] with command [[rm -rf /data/dist]]
08:26:36 BIOCONDA INFO (ERR) [Jun  4 08:26:36] DEBU Creating container [step-2a334538db]
08:26:36 BIOCONDA INFO (ERR) [Jun  4 08:26:36] DEBU Image [quay.io/bioconda/create-env:latest] not present, pulling it
08:26:36 BIOCONDA INFO (ERR) [Jun  4 08:26:36] DEBU Pull Image [quay.io/bioconda/create-env:latest]
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] DEBU Created container [bd5389edf3ca step-2a334538db], starting it
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] DEBU Container [bd5389edf3ca step-2a334538db] started, waiting for completion
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] DEBU Container [bd5389edf3ca step-2a334538db] completed with exit code [0] as expected
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] DEBU Container [bd5389edf3ca step-2a334538db] removed
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] STEP Run image [quay.io/bioconda/create-env:latest] with command [[/bin/sh -c conda install  -c 'conda-forge' -c 'file:///home/debian/anaconda3/envs/bioconda/conda-bld' -c 'bioconda'  'taxidtools=2.3.0=pyhdfd78af_0' --strict-channel-priority -p /usr/local --copy --yes --quiet&&create-env --conda=: /usr/local]]
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] DEBU Creating container [step-62d30140fc]
08:26:48 BIOCONDA INFO (ERR) [Jun  4 08:26:48] DEBU Created container [6cc58ec48140 step-62d30140fc], starting it
08:26:49 BIOCONDA INFO (ERR) [Jun  4 08:26:49] DEBU Container [6cc58ec48140 step-62d30140fc] started, waiting for completion
08:26:50 BIOCONDA INFO (ERR) [Jun  4 08:26:50] SOUT Channels:
08:26:50 BIOCONDA INFO (ERR) [Jun  4 08:26:50] SOUT - conda-forge
08:26:50 BIOCONDA INFO (ERR) [Jun  4 08:26:50] SOUT - file:///home/debian/anaconda3/envs/bioconda/conda-bld
08:26:50 BIOCONDA INFO (ERR) [Jun  4 08:26:50] SOUT - bioconda
08:26:50 BIOCONDA INFO (ERR) [Jun  4 08:26:50] SOUT Platform: linux-64
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SOUT Collecting package metadata (repodata.json): ...working... done
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SOUT Solving environment: ...working... failed
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SERR
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SERR LibMambaUnsatisfiableError: Encountered problems while solving:
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SERR - package taxidtools-2.3.0-pyhdfd78af_0 is excluded by strict repo priority
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SERR
08:27:04 BIOCONDA INFO (ERR) [Jun  4 08:27:04] SERR
08:27:05 BIOCONDA INFO (ERR) [Jun  4 08:27:05] ERRO Task processing failed: Unexpected exit code [1] of container [6cc58ec48140 step-62d30140fc], container preserved
08:27:05 BIOCONDA INFO (OUT) .

Now it fails because of an older release in conda-forge but that's my problem to deal with.