search

bioconda / bioconda-utils

Utilities for building and managing bioconda recipes
MIT License
98 stars 133 forks source link

build is adding unexpected 'placehold' paths and failing to find files on aarch64 #1015

Closed dslarm closed 2 weeks ago

dslarm commented 4 weeks ago

@aliciaaevans @bgruening @FriederikeHanssen In this pull-request for a relatively simple recipe, aarch64 builds are failing in the CI for the pull request.

They work locally in a bog-standard Rocky Linux aarch64 system set up per the instructions on building locally.

Specifically, in the recipe, it fails to find a perl script that is part of the package and build.sh - but the path is given as some large sequence of placehold_ strings.

11:37:11 BIOCONDA INFO (OUT) + which get_fasta_lengths.pl
11:37:11 BIOCONDA INFO (OUT)     -v         verbose mode
11:37:11 BIOCONDA INFO (OUT) 
11:37:11 BIOCONDA INFO (OUT) which: no get_fasta_lengths.pl in (/opt/conda/conda-bld/control-freec_1730288067229/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac:/opt/conda/conda-bld/control-freec_1730288067229/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin:/opt/conda/conda-bld/control-freec_1730288067229/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/conda/bin)
11:37:12 BIOCONDA INFO (OUT) WARNING: Tests failed for control-freec-11.6b-h7021222_1.tar.bz2 - moving package to /opt/conda/conda-bld/broken

Any thoughts?

rpetit3 commented 4 weeks ago

Hi @dslarm

Could you provide a link to the CI? I'm not seeing control-freec in the link you provided above.

Cheers, Robert

EDIT:

https://github.com/bioconda/bioconda-recipes/pull/51766 https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/132263/workflows/24564e32-0469-410a-a819-444d8475e930/jobs/216554?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

dslarm commented 3 weeks ago

I've dug into the logs on this a bit more. I just commented to the original ticket for control-freec where I read the logs (https://github.com/bioconda/bioconda-recipes/pull/51766)

The profusion of 'placehold' path components in PREFIX seems to be common and partially the issue: the paths appear to be being truncated (on aarch64 but not on x86_64 - perhaps build environments/shells are different?).

The cp copies to a path ending _placehold/bin on both platforms - but note the search path returned:

On linux-aarch64 has only one placehold ... bin/ in the PATH of which but the last part of it ends in _plac/bin - not _placehold/bin.. On linux-x86_64, the $PREFIX/bin is where the perl script is found, and PREFIX itself end with _placehold. Bad: on linux-aarch64

14:36:32  (OUT) export PREFIX=/opt/conda/conda-bld/control-freec_1730298923242/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold�[0m
[..]
14:36:33  (OUT) + chmod +x scripts/get_fasta_lengths.pl�[0m
[..]
14:36:33  (OUT) + cp scripts/get_fasta_lengths.pl /opt/conda/conda-bld/control-freec_1730298923242/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin�[0m
[..]
14:38:14  (OUT) + which get_fasta_lengths.pl�[0m
14:38:14  (OUT) which: no get_fasta_lengths.pl in (/opt/conda/conda-bld/control-freec_1730298923242/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac:/opt/conda/conda-bld/control-freec_1730298923242/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin:/opt/conda/conda-bld/control-freec_1730298923242/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/conda/bin)�[0m

Good: on linux-x86_64

14:40:04.0285210Z  (OUT) export PREFIX=/opt/conda/conda-bld/control-freec_1730299114023/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold�[0m
[..]
14:40:04.0753832Z  (OUT) + chmod +x scripts/get_fasta_lengths.pl�[0m
[..]
14:40:04.0763229Z  (OUT) + cp scripts/get_fasta_lengths.pl /opt/conda/conda-bld/control-freec_1730299114023/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold/bin�[0m
[..]
14:41:30.0185472Z  (OUT) + which get_fasta_lengths.pl�[0m
14:41:30.0186581Z  (OUT) $PREFIX/bin/get_fasta_lengths.pl�[0m...
dslarm commented 2 weeks ago

I've read a bit more about the placehold ( https://github.com/conda/conda-build/issues/1482 in case anyone stumbles on this again) - and this is not the issue but is correct behaviour.

What appears to be happening is that in the first phase, the package is built:

Then, the second phase, testing the build:

Quite why the files are not found at the unpacked test phase, I don't know. I think a bit of debugging of the package can be tried to see if they are copied over successfully.

I'll close this for now and reopen if it proves to be in the utils and not the package.

dslarm commented 2 weeks ago

@rpetit3 all, I have this one figured out.

The "which" command in RHEL/Centos 7 prefers full paths to be < 256 characters. That version of which is failing for the long 'placehold' paths on aarch64 because the directory itself is over 250 chars already, and so adding files > 15-20 chars name length or so will trigger the failure in which.

On x86 it is not failing on the control-freec examples on account of luck rather than of design.

Redhat did a fix in RHEL-9, also applied in Rocky 9.

Ubuntu uses a different 'which' implementation - that is why local build testing outside of a container worked for me.

The best solutions would be updating the build / test containers to Rocky 9, or backport the which source fix, but in the meanwhile, there are a number of recipes with long script/binary names and that test using "which" that are not yet enabled for linux-aarch64 and that will trip over this issue if they are ported.

FWIW, the patch backport - if that's the chosen route - can be found in https://download.rockylinux.org/pub/rocky/9/BaseOS/source/tree/Packages/w/which-2.21-29.el9.src.rpm as the file which-2.21-path_max.patch.

martin-g commented 1 week ago

@dslarm Maybe you should use https://anaconda.org/conda-forge/which ?! AFAIK there is a work on updating the Docker images used for building from CentOS 7 to AlmaLinux 9.x at Conda-forge. I guess this work will be used/applied on BioConda too at some point.

dslarm commented 1 week ago

Thanks Martin - I didn’t know about that. That needs the patch .. ticket opened: https://github.com/conda-forge/which-feedstock/issues/5