search

bioconda / bioconda-recipes

Conda recipes for the bioconda channel.
https://bioconda.github.io
MIT License
1.65k stars 3.29k forks source link

Different glibc version between aarch64 and x86 builders? #46186

Open corneliusroemer opened 9 months ago

corneliusroemer commented 9 months ago

I've tried to add aarch64 support to the nextclade recipe here #46185 which does binary repackaging.

I noticed that the aarch64 tests fail with:

15:17:28 BIOCONDA INFO (OUT) + nextclade --version
15:17:28 BIOCONDA INFO (OUT) nextclade: relocation error: nextclade: symbol __cxa_thread_atexit_impl, version GLIBC_2.18 not defined in file libc.so.6 with link time reference

The x86 tests don't fail - suggesting the version of glibc that comes with the aarch64 builder is lower than the one that comes with x86. Is that correct?

I know we should build from source - but repackaging has worked flawlessly to date. Others might run into the same issue.

martin-g commented 9 months ago

Both Docker images (quay.io/bioconda/bioconda-utils-build-env-cos7:latest and quay.io/bioconda/bioconda-utils-build-env-cos7-aarch64:latest) are based on CentOS 7 (7.9.2009)

aarch64:

[conda@26d3c04003f9 ~]$ ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[conda@26d3c04003f9 ~]$ ld --version
GNU ld version 2.27-44.base.el7_9.1
Copyright (C) 2016 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

strings /usr/lib64/libc.so.6 | grep GLIBC
GLIBC_2.17
GLIBC_2.18
GLIBC_PRIVATE
gethostbyname2_r@@GLIBC_2.17
_IO_fopen@@GLIBC_2.17
_IO_file_finish@@GLIBC_2.17
_dl_starting_up@@GLIBC_PRIVATE
_IO_fgetpos@@GLIBC_2.17
getaliasent_r@@GLIBC_2.17
_IO_file_seekoff@@GLIBC_2.17
getrpcent_r@@GLIBC_2.17
getservbyname_r@@GLIBC_2.17
_rtld_global@@GLIBC_PRIVATE
_IO_file_init@@GLIBC_2.17
getservbyport_r@@GLIBC_2.17
__libc_enable_secure@@GLIBC_PRIVATE
posix_spawn@@GLIBC_2.17
tmpfile@@GLIBC_2.17
realpath@@GLIBC_2.17
_IO_fgetpos64@@GLIBC_2.17
getrpcbyname_r@@GLIBC_2.17
_rtld_global_ro@@GLIBC_PRIVATE
__pointer_chk_guard@@GLIBC_PRIVATE
getspent_r@@GLIBC_2.17
_IO_file_close_it@@GLIBC_2.17
sys_sigabbrev@@GLIBC_2.17
_IO_fsetpos@@GLIBC_2.17
_sys_nerr@@GLIBC_2.17
pthread_cond_timedwait@@GLIBC_2.17
_IO_do_write@@GLIBC_2.17
getnetbyname_r@@GLIBC_2.17
getpwnam_r@@GLIBC_2.17
_mcount@@GLIBC_2.18
gethostbyname_r@@GLIBC_2.17
_IO_file_setbuf@@GLIBC_2.17
pthread_cond_destroy@@GLIBC_2.17
getaliasbyname_r@@GLIBC_2.17
getprotoent_r@@GLIBC_2.17
getspnam_r@@GLIBC_2.17
getrpcbynumber_r@@GLIBC_2.17
nftw64@@GLIBC_2.17
getnetent_r@@GLIBC_2.17
_res@GLIBC_2.17
posix_spawnp@@GLIBC_2.17
_IO_popen@@GLIBC_2.17
pclose@@GLIBC_2.17
nftw@@GLIBC_2.17
_IO_file_xsputn@@GLIBC_2.17
_sys_errlist@@GLIBC_2.17
_IO_proc_open@@GLIBC_2.17
_IO_file_overflow@@GLIBC_2.17
_IO_fdopen@@GLIBC_2.17
getgrgid_r@@GLIBC_2.17
getnetbyaddr_r@@GLIBC_2.17
__res_maybe_init@GLIBC_PRIVATE
pthread_cond_signal@@GLIBC_2.17
h_errno@GLIBC_PRIVATE
_IO_file_attach@@GLIBC_2.17
_IO_file_sync@@GLIBC_2.17
getpwuid_r@@GLIBC_2.17
pthread_cond_wait@@GLIBC_2.17
_sys_siglist@@GLIBC_2.17
fopencookie@@GLIBC_2.17
pthread_cond_broadcast@@GLIBC_2.17
getpwent_r@@GLIBC_2.17
gethostbyaddr_r@@GLIBC_2.17
sched_setaffinity@@GLIBC_2.17
getgrent_r@@GLIBC_2.17
fnmatch@@GLIBC_2.17
getgrnam_r@@GLIBC_2.17
_IO_fclose@@GLIBC_2.17
pthread_cond_init@@GLIBC_2.17
regexec@@GLIBC_2.17
localeconv@@GLIBC_2.17
_IO_file_write@@GLIBC_2.17
gethostent_r@@GLIBC_2.17
_IO_proc_close@@GLIBC_2.17
getservent_r@@GLIBC_2.17
_IO_file_fopen@@GLIBC_2.17
getprotobyname_r@@GLIBC_2.17
sched_getaffinity@@GLIBC_2.17
__tls_get_addr@@GLIBC_2.17
getprotobynumber_r@@GLIBC_2.17
_IO_file_underflow@@GLIBC_2.17
_dl_argv@@GLIBC_PRIVATE
pthread_attr_init@@GLIBC_2.17
_IO_fsetpos64@@GLIBC_2.17

x86_64:

[conda@96dbbda9d9fa ~]$ ldd --version
ldd (GNU libc) 2.17
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

[conda@96dbbda9d9fa ~]$ ld --version
GNU ld version 2.27-44.base.el7_9.1
Copyright (C) 2016 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

strings /usr/lib64/libc.so.6 | grep GLIBC
GLIBC_2.2.5
GLIBC_2.2.6
GLIBC_2.3
GLIBC_2.3.2
GLIBC_2.3.3
GLIBC_2.3.4
GLIBC_2.4
GLIBC_2.5
GLIBC_2.6
GLIBC_2.7
GLIBC_2.8
GLIBC_2.9
GLIBC_2.10
GLIBC_2.11
GLIBC_2.12
GLIBC_2.13
GLIBC_2.14
GLIBC_2.15
GLIBC_2.16
GLIBC_2.17
GLIBC_PRIVATE
___sys_nerr_GLIBC_2_3
___sys_nerr_GLIBC_2_4
___sys_nerr_GLIBC_2_1
___sys_errlist_GLIBC_2_1
___sys_errlist_GLIBC_2_4
___sys_errlist_GLIBC_2_3
_IO_file_seekoff@@GLIBC_2.2.5
getservent_r@@GLIBC_2.2.5
_sys_siglist@@GLIBC_2.3.3
_dl_starting_up@@GLIBC_PRIVATE
tmpfile@@GLIBC_2.2.5
sched_getaffinity@GLIBC_2.3.3
pthread_cond_wait@GLIBC_2.2.5
regexec@GLIBC_2.2.5
memcpy@@GLIBC_2.14
_IO_file_init@@GLIBC_2.2.5
realpath@GLIBC_2.2.5
realpath@@GLIBC_2.3
__ctype32_tolower@GLIBC_2.2.5
gethostbyname2_r@@GLIBC_2.2.5
getaliasbyname_r@@GLIBC_2.2.5
_IO_file_overflow@@GLIBC_2.2.5
pthread_cond_wait@@GLIBC_2.3.2
getprotoent_r@@GLIBC_2.2.5
regexec@@GLIBC_2.3.4
_rtld_global@@GLIBC_PRIVATE
getgrgid_r@@GLIBC_2.2.5
pthread_cond_init@GLIBC_2.2.5
_IO_fsetpos@@GLIBC_2.2.5
GLIBC_2.8
_IO_proc_close@@GLIBC_2.2.5
_sys_nerr@@GLIBC_2.12
pthread_cond_timedwait@@GLIBC_2.3.2
__libc_enable_secure@@GLIBC_PRIVATE
pclose@@GLIBC_2.2.5
getaliasent_r@@GLIBC_2.2.5
__tls_get_addr@@GLIBC_2.3
_sys_nerr@GLIBC_2.4
sys_sigabbrev@GLIBC_2.2.5
pthread_cond_broadcast@GLIBC_2.2.5
_IO_proc_open@@GLIBC_2.2.5
posix_spawn@@GLIBC_2.15
getpwnam_r@@GLIBC_2.2.5
_rtld_global_ro@@GLIBC_PRIVATE
getrpcbyname_r@@GLIBC_2.2.5
pthread_cond_broadcast@@GLIBC_2.3.2
nftw64@GLIBC_2.2.5
_sys_siglist@GLIBC_2.2.5
_sys_nerr@GLIBC_2.3
_IO_fsetpos64@@GLIBC_2.2.5
__ctype_tolower@GLIBC_2.2.5
__ctype_toupper@GLIBC_2.2.5
_IO_file_underflow@@GLIBC_2.2.5
localeconv@@GLIBC_2.2.5
pthread_cond_timedwait@GLIBC_2.2.5
_IO_file_fopen@@GLIBC_2.2.5
__ctype_b@GLIBC_2.2.5
GLIBC_2.5
_IO_file_finish@@GLIBC_2.2.5
GLIBC_2.9
getgrnam_r@@GLIBC_2.2.5
GLIBC_2.7
GLIBC_2.6
getservbyname_r@@GLIBC_2.2.5
__ctype32_b@GLIBC_2.2.5
_res@GLIBC_2.2.5
__secure_getenv@GLIBC_2.2.5
getnetent_r@@GLIBC_2.2.5
nftw@GLIBC_2.2.5
getpwuid_r@@GLIBC_2.2.5
_IO_fdopen@@GLIBC_2.2.5
posix_spawn@GLIBC_2.2.5
gethostbyaddr_r@@GLIBC_2.2.5
_IO_file_xsputn@@GLIBC_2.2.5
pthread_attr_init@@GLIBC_2.2.5
getrpcbynumber_r@@GLIBC_2.2.5
_IO_fopen@@GLIBC_2.2.5
getnetbyname_r@@GLIBC_2.2.5
memcpy@GLIBC_2.2.5
pthread_cond_signal@@GLIBC_2.3.2
pthread_cond_signal@GLIBC_2.2.5
sys_sigabbrev@@GLIBC_2.3.3
_IO_fgetpos@@GLIBC_2.2.5
posix_spawnp@@GLIBC_2.15
_IO_file_setbuf@@GLIBC_2.2.5
getservbyport_r@@GLIBC_2.2.5
_sys_errlist@GLIBC_2.3
_IO_file_attach@@GLIBC_2.2.5
gethostent_r@@GLIBC_2.2.5
_sys_errlist@GLIBC_2.4
_sys_nerr@GLIBC_2.2.5
nftw64@@GLIBC_2.3.3
_IO_fgetpos64@@GLIBC_2.2.5
__res_maybe_init@GLIBC_PRIVATE
fnmatch@@GLIBC_2.2.5
h_errno@GLIBC_PRIVATE
_sys_errlist@@GLIBC_2.12
_IO_fclose@@GLIBC_2.2.5
gethostbyname_r@@GLIBC_2.2.5
_IO_do_write@@GLIBC_2.2.5
getspent_r@@GLIBC_2.2.5
getpwent_r@@GLIBC_2.2.5
pthread_cond_destroy@GLIBC_2.2.5
sched_setaffinity@GLIBC_2.3.3
getspnam_r@@GLIBC_2.2.5
sched_setaffinity@@GLIBC_2.3.4
_IO_popen@@GLIBC_2.2.5
getgrent_r@@GLIBC_2.2.5
_IO_file_sync@@GLIBC_2.2.5
GLIBC_2.11
GLIBC_2.16
GLIBC_2.10
pthread_cond_init@@GLIBC_2.3.2
GLIBC_2.17
getprotobyname_r@@GLIBC_2.2.5
GLIBC_2.13
getnetbyaddr_r@@GLIBC_2.2.5
_sys_errlist@GLIBC_2.2.5
_IO_file_write@@GLIBC_2.2.5
pthread_cond_destroy@@GLIBC_2.3.2
nftw@@GLIBC_2.3.3
GLIBC_2.2.6
fopencookie@@GLIBC_2.2.5
__ctype32_toupper@GLIBC_2.2.5
sched_getaffinity@@GLIBC_2.3.4
_dl_argv@@GLIBC_PRIVATE
posix_spawnp@GLIBC_2.2.5
getrpcent_r@@GLIBC_2.2.5
getprotobynumber_r@@GLIBC_2.2.5
_IO_file_close_it@@GLIBC_2.2.5
_obstack@GLIBC_2.2.5
corneliusroemer commented 9 months ago

Thanks @martin-g! That's quite an old version and current Rust versions require at least glibc 2.17 Rust announcement which should be fine. However for some reason sometimes glibc 2.18 is in fact required, see blog post

Trying to see what the difference between aarch and x86 is, I noticed that aarch has the following entry which x86 doesn't:

aarch:

$ strings /usr/lib64/libc.so.6 | grep GLIBC_2.18
GLIBC_2.18

x86:

$ strings /usr/lib64/libc.so.6 | grep GLIBC_2.18

Could this be related to it? What do you suggest we do in this case? Switch off tests and wait for builders to use more modern centos? We could package the musl build, but that's got terrible performance for multi-threading, an important feature of Nextclade.

See Rust issue: https://github.com/rust-lang/rust/issues/57497

Centos7 will reach end of life in June 2024: https://www.redhat.com/en/events/webinar/centos-linux-reaching-its-end-life-now-what

jmarshall commented 9 months ago

The glibc version is controlled by the sysroot_linux-ARCH package pulled in by conda-build. For us, it is specified by conda-forge-pinning's _conda_buildconfig.yaml as described here and here:

c_stdlib_version:              # [unix]
  - 2.12                       # [linux and x86_64]
  - 2.17                       # [linux and not x86_64]

(Your checking of /usr/lib64/libc.so.6 versions above appears to be checking the OS-installed glibc rather than a conda-supplied one?!)

It is indeed annoying that these are so old. I had to patch htslib to add linux-aarch64 support to work around the ancient 2.17 to avoid https://github.com/samtools/htscodecs/issues/88#issuecomment-1664421461 and now I finally understand why conda builds choose this ancient version!

The first “here” link in my first paragraph shows how to change the default by adding your own _conda_buildconfig.yaml to your recipe. This may or may not have undesired side-effects, and may not be relevant for your binary-copying recipe. I used a different approach for htslib before I found that documentation.

Updating these is conda-forge/conda-forge.github.io#1941 but would seem to be a long way off — see the discussion there and e.g. https://github.com/conda-forge/ctng-compilers-feedstock/issues/109#issuecomment-1960473778.

corneliusroemer commented 8 months ago

Thanks @jmarshall, that's very insightful! So the glibc versions do seem to be different after all - which may or may not explain why I see failure only in aarch64 and not in arm64.

I'll try to change the build config to see if that fixes it.

Update: Unfortunately setting sysroot_linux-aarch64 2.28.* in the hope of getting a builder with __glibc >=2.28 doesn't work as there doesn't seem to be a builder available with such a minimum glibc version. Shame.

21:11:23 BIOCONDA INFO (OUT) RuntimeError: Solver could not find solution.Mamba failed to solve:
21:11:23 BIOCONDA INFO (OUT)  - __glibc >=2.28,<3.0.a0
21:11:23 BIOCONDA INFO (OUT)  - sysroot_linux-aarch64 2.28.*

https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/110243/workflows/238b4621-96e8-4725-b174-553c497413ba/jobs/181068

Other recipes that repackage binaries and mention glibc in the context of having build/test issues often resort to disabling tests. I guess that's the workaround for now:

martin-g commented 8 months ago

https://github.com/bioconda/bioconda-recipes/pull/46341 is a similar issue. Is it correct to say that the tests try to use the OS-installed glibc ? Is there a way to preload the sysroot_linux-ARCH one ?