=sci-mathematics/sagemath-standard-10.3 fails as: `AttributeError: Can't pickle local object '_prepare_extension_detection.<locals>.<lambda>'`

[trofi]

Tried to build sagemath-standard to reproduce a possible tool chain failure. Got the build failure as:

# emerge -av1 sagemath-standard

``` # emerge -av1 sagemath-standard ... [ebuild N ] sci-mathematics/sagemath-standard-10.3::sage-on-gentoo USE="X doc jmol -debug -latex -test" PYTHON_TARGETS="python3_11 -python3_10 -python3_12" 0 KiB [ebuild N ] sci-mathematics/sage-doc-10.3::sage-on-gentoo USE="-doc-pdf" L10N="-ca -de -es -fr -hu -it -ja -pt -ru -tr" 0 KiB ... ************************************************************************ Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/lib/python3.11/site-packages/gpep517/__main__.py", line 429, in sys.exit(main()) ^^^^^^ File "/usr/lib/python3.11/site-packages/gpep517/__main__.py", line 425, in main return func(args) ^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/gpep517/__main__.py", line 215, in build_wheel print(build_wheel_impl(args, args.wheel_dir), file=out) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/gpep517/__main__.py", line 207, in build_wheel_impl wheel_name = backend.build_wheel(str(wheel_dir), args.config_json) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/setuptools/build_meta.py", line 410, in build_wheel return self._build_with_temp_dir( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir self.run_setup() File "/usr/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup exec(code, locals()) File "", line 105, in File "/usr/lib/python3.11/site-packages/setuptools/__init__.py", line 104, in setup return distutils.core.setup(**attrs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup return run_commands(dist) ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands dist.run_commands() File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command super().run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run self.run_command("build") File "/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command self.distribution.run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command super().run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run self.run_command(cmd_name) File "/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command self.distribution.run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command super().run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/lib/python3.11/site-packages/sage_setup/command/sage_build_ext.py", line 25, in run self.run_command('build_cython') File "/usr/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command self.distribution.run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command super().run_command(command) File "/usr/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/usr/lib/python3.11/site-packages/sage_setup/command/sage_build_cython.py", line 220, in run extensions = cythonize( ^^^^^^^^^^ File "/usr/lib/python3.11/site-packages/Cython/Build/Dependencies.py", line 1150, in cythonize result.get(99999) # seconds ^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/multiprocessing/pool.py", line 774, in get raise self._value File "/usr/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks put(task) File "/usr/lib/python3.11/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps cls(buf, protocol).dump(obj) AttributeError: Can't pickle local object '_prepare_extension_detection..' ************************************************************************ Error building the Sage library ************************************************************************ * ERROR: sci-mathematics/sagemath-standard-10.3::sage-on-gentoo failed (compile phase): * Wheel build failed * * Call stack: * ebuild.sh, line 136: Called src_compile * environment, line 4476: Called distutils-r1_src_compile * environment, line 1978: Called _distutils-r1_run_foreach_impl 'distutils-r1_python_compile' * environment, line 731: Called python_foreach_impl 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 4053: Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 3491: Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 3489: Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 1294: Called distutils-r1_run_phase 'distutils-r1_python_compile' * environment, line 1960: Called distutils-r1_python_compile * environment, line 1776: Called distutils_pep517_install '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/work/sagemath-standard-10.3-python3_11/install' * environment, line 2345: Called die * The specific snippet of code: * local wheel=$("${cmd[@]}" 3>&1 1>&2 || die "Wheel build failed"); * * If you need support, post the output of `emerge --info '=sci-mathematics/sagemath-standard-10.3::sage-on-gentoo'`, * the complete build log and the output of `emerge -pqv '=sci-mathematics/sagemath-standard-10.3::sage-on-gentoo'`. * The complete build log is located at '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/temp/build.log'. * The ebuild environment file is located at '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/temp/environment'. * Working directory: '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/work/sagemath-standard-10.3' * S: '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/work/sagemath-standard-10.3' >>> Failed to emerge sci-mathematics/sagemath-standard-10.3 * Messages for package sci-mathematics/sagemath-standard-10.3: * ERROR: sci-mathematics/sagemath-standard-10.3::sage-on-gentoo failed (compile phase): * Wheel build failed * * Call stack: * ebuild.sh, line 136: Called src_compile * environment, line 4476: Called distutils-r1_src_compile * environment, line 1978: Called _distutils-r1_run_foreach_impl 'distutils-r1_python_compile' * environment, line 731: Called python_foreach_impl 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 4053: Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 3491: Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 3489: Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'distutils-r1_python_compile' * environment, line 1294: Called distutils-r1_run_phase 'distutils-r1_python_compile' * environment, line 1960: Called distutils-r1_python_compile * environment, line 1776: Called distutils_pep517_install '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/work/sagemath-standard-10.3-python3_11/install' * environment, line 2345: Called die * The specific snippet of code: * local wheel=$("${cmd[@]}" 3>&1 1>&2 || die "Wheel build failed"); * * If you need support, post the output of `emerge --info '=sci-mathematics/sagemath-standard-10.3::sage-on-gentoo'`, * the complete build log and the output of `emerge -pqv '=sci-mathematics/sagemath-standard-10.3::sage-on-gentoo'`. * The complete build log is located at '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/temp/build.log'. * The ebuild environment file is located at '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/temp/environment'. * Working directory: '/dev/shm/portage/sci-mathematics/sagemath-standard-10.3/work/sagemath-standard-10.3' ```

# emerge --info

``` # emerge --info Portage 3.0.64 (python 3.11.9-final-0, default/linux/amd64/17.1, gcc-13, glibc-2.39-r4, 6.9.0-rc5 x86_64) ================================================================= System uname: Linux-6.9.0-rc5-x86_64-AMD_Ryzen_9_5950X_16-Core_Processor-with-glibc2.39 KiB Mem: 131848004 total, 17565464 free KiB Swap: 197770748 total, 195366652 free Head commit of repository gentoo: f653248b062439bfc6fafabae43cb1b94952ab1d Head commit of repository nix-guix: c91980e8812b78b0221642f5edaf697d479961ab Head commit of repository sage-on-gentoo: 1669f0e889ca9951f26aafdd33558d9590cc6618 sh bash 5.2_p26-r3 ld GNU ld (Gentoo 2.41 p5) 2.41.0 ccache version 4.9.1 [disabled] app-misc/pax-utils: 1.3.7::gentoo app-shells/bash: 5.2_p26-r3::gentoo dev-build/autoconf: 2.72-r1::gentoo dev-build/automake: 1.16.5-r2::gentoo dev-build/cmake: 3.29.2::gentoo dev-build/libtool: 2.4.7-r4::gentoo dev-build/make: 4.4.1-r1::gentoo dev-build/meson: 1.4.0-r1::gentoo dev-java/java-config: 2.3.3-r1::gentoo dev-lang/perl: 5.38.2-r2::gentoo dev-lang/python: 3.11.9::gentoo, 3.12.3::gentoo dev-lang/rust: 1.77.1::gentoo dev-util/ccache: 4.9.1-r1::gentoo sys-apps/baselayout: 2.15::gentoo sys-apps/sandbox: 2.38::gentoo sys-apps/systemd: 254.11::gentoo sys-devel/binutils: 2.41-r5::gentoo, 2.42-r1::gentoo sys-devel/binutils-config: 5.5::gentoo sys-devel/gcc: 12.3.1_p20240209::gentoo, 13.2.1_p20240503::gentoo sys-devel/gcc-config: 2.11::gentoo sys-devel/llvm: 17.0.6::gentoo, 18.1.5::gentoo sys-kernel/linux-headers: 6.8-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.39-r4::gentoo Repositories: gentoo location: /var/db/repos/gentoo sync-type: git sync-uri: https://github.com/gentoo/gentoo.git priority: -1000 volatile: False crossdev location: /co masters: gentoo volatile: True nix-guix location: /var/db/repos/nix-guix sync-type: git sync-uri: https://github.com/trofi/nix-guix-gentoo.git masters: gentoo volatile: False sage-on-gentoo location: /var/db/repos/sage-on-gentoo sync-type: git sync-uri: https://github.com/cschwan/sage-on-gentoo masters: gentoo volatile: False ACCEPT_KEYWORDS="amd64 ~amd64" ACCEPT_LICENSE="@FREE" CBUILD="x86_64-pc-linux-gnu" CFLAGS="-O2 -pipe -fdiagnostics-show-option -frecord-gcc-switches" CHOST="x86_64-pc-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo /etc/texmf/language.dat.d /etc/texmf/language.def.d /etc/texmf/updmap.d /etc/texmf/web2c" CXXFLAGS="-O2 -pipe" DISTDIR="/bound/distfiles" ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs buildpkg-live config-protect-if-modified distlocks ebuild-locks fail-clean fixlafiles ipc-sandbox merge-sync merge-wait multilib-strict network-sandbox news parallel-fetch pid-sandbox pkgdir-index-trusted preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://distfiles.gentoo.org" LANG="C.UTF8" LDFLAGS="-Wl,-O1 -Wl,--as-needed -Wl,--hash-style=gnu" LEX="flex" MAKEOPTS="-j17 -l17" PKGDIR="/var/cache/binpkgs" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/dev/shm" SHELL="/bin/bash" USE="X acl amd64 bzip2 cli crypt dri fortran gdbm harfbuzz iconv ipv6 libtirpc minizip multilib ncurses nls opengl openmp pam pcre readline seccomp split-usr ssl systemd test-rust text unicode xattr zlib" ABI_X86="64" ADA_TARGET="gcc_12" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 ntrip navcom oceanserver oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 tsip tripmate tnt ublox" INPUT_DEVICES="libinput" KERNEL="linux" LCD_DEVICES="bayrad cfontz glk hd44780 lb216 lcdm001 mtxorb text" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php8-1" POSTGRES_TARGETS="postgres15" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby31" VIDEO_CARDS="amdgpu fbdev intel nouveau radeon radeonsi vesa dummy" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipp2p iface geoip fuzzy condition tarpit sysrq proto logmark ipmark dhcpmac delude chaos account" Unset: ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, PYTHONPATH, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS ```

[trofi]

Tl;DR: If I uninstall it with emerge -C dev-python/scikit-build-core I am able to install sci-mathematics/sagemath-standard.

More words:

Looks like _prepare_extension_detection is defined in dev-python/scikit-build-core. It injects a lambda as https://github.com/scikit-build/scikit-build-core/blob/f6ed5a28fc85e621b03d984011d17def888ee0db/src/scikit_build_core/setuptools/build_cmake.py#L179:

def _prepare_extension_detection(dist: Distribution) -> None:
    # Setuptools needs to know that it has extensions modules

    orig_has_ext_modules = dist.has_ext_modules
    dist.has_ext_modules = lambda: orig_has_ext_modules() or _has_cmake(dist)  # type: ignore[method-assign]
    # ...

which fails to pickle/unpickle across process boundaries somewhere in the build system.

It does not look like sci-mathematics/sagemath-standard needs it. If I uninstall it with emerge -C dev-python/scikit-build-core I am able to install sci-mathematics/sagemath-standard. The only way I found it is by grepping /usr/lib for _prepare_extension_detection.

[kiwifb]

Hum... This is something I may take upstream if I can reproduce it with vanilla sage. This is quite the intrusive behavior.

[kiwifb]

class EvilList in the body of that function, https://github.com/scikit-build/scikit-build-core/blob/f6ed5a28fc85e621b03d984011d17def888ee0db/src/scikit_build_core/setuptools/build_cmake.py#L188 you cannot make this kind of thing up.

[kiwifb]

Just got bitten. It is pulled via pandas[full-support] which pulls blosc which in turns wants scikit-build :P I am expecting other packages will get impacted at some point. sage itself is supposed to move to meson as a build system in the not so distant future, that will help.

[strogdon]

As well here

AttributeError: Can't pickle local object '_prepare_extension_detection.<locals>.<lambda>'
************************************************************************
Error building the Sage library
************************************************************************
 * ERROR: sci-mathematics/sagemath-standard-9999::sage-on-gentoo failed (compile phase):
 *   Wheel build failed
 * 
 * Call stack:
 *     ebuild.sh, line  136:  Called src_compile
 *   environment, line 5241:  Called distutils-r1_src_compile
 *   environment, line 2347:  Called _distutils-r1_run_foreach_impl 'distutils-r1_python_compile'
 *   environment, line  932:  Called python_foreach_impl 'distutils-r1_run_phase' 'distutils-r1_python_compile'
 *   environment, line 4791:  Called multibuild_foreach_variant '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'distutils-r1_python_compile'
 *   environment, line 4276:  Called _multibuild_run '_python_multibuild_wrapper' 'distutils-r1_run_phase' 'distutils-r1_python_compile'
 *   environment, line 4274:  Called _python_multibuild_wrapper 'distutils-r1_run_phase' 'distutils-r1_python_compile'
 *   environment, line 1649:  Called distutils-r1_run_phase 'distutils-r1_python_compile'
 *   environment, line 2329:  Called distutils-r1_python_compile
 *   environment, line 2144:  Called distutils_pep517_install '/var/tmp/portage/sci-mathematics/sagemath-standard-9999/work/sagemath-standard-9999-python3_11/install'
 *   environment, line 2729:  Called die
 * The specific snippet of code:
 *       [[ -n ${wheel} ]] || die "No wheel name returned";

when building 10.4.beta8.

[strogdon]

This happened after switching system python to python3_12 and rebuilding sagemath pulled in scikit-core.

[kiwifb]

That's surprising. What exactly pulls scikit-core? For me, ultimately it was coming from pandas.

[strogdon]

Not sure. I do not have pandas installed. Here is a portion of the emerge log

1717392020:  >>> emerge (155 of 237) dev-python/scikit-build-core-0.9.4 to /
1717392020:  === (155 of 237) Cleaning (dev-python/scikit-build-core-0.9.4::/var/db/repos/gentoo/dev-python/scikit-build-core/scikit-build-core-0.9.4.ebuild)
1717392020:  === (155 of 237) Compiling/Merging (dev-python/scikit-build-core-0.9.4::/var/db/repos/gentoo/dev-python/scikit-build-core/scikit-build-core-0.9.4.ebuild)
1717392027:  === (155 of 237) Merging (dev-python/scikit-build-core-0.9.4::/var/db/repos/gentoo/dev-python/scikit-build-core/scikit-build-core-0.9.4.ebuild)
1717392028:  >>> AUTOCLEAN: dev-python/scikit-build-core:0
1717392028:  === Unmerging... (dev-python/scikit-build-core-0.9.4)
1717392029:  >>> unmerge success: dev-python/scikit-build-core-0.9.4
1717392031:  === (155 of 237) Post-Build Cleaning (dev-python/scikit-build-core-0.9.4::/var/db/repos/gentoo/dev-python/scikit-build-core/scikit-build-core-0.9.4.ebuild)
1717392031:  ::: completed emerge (155 of 237) dev-python/scikit-build-core-0.9.4 to /
1717392031:  >>> emerge (156 of 237) sci-mathematics/primesieve-11.0 to /
1717392031:  === (156 of 237) Cleaning (sci-mathematics/primesieve-11.0::/var/db/repos/gentoo/sci-mathematics/primesieve/primesieve-11.0.ebuild)
1717392031:  === (156 of 237) Compiling/Merging (sci-mathematics/primesieve-11.0::/var/db/repos/gentoo/sci-mathematics/primesieve/primesieve-11.0.ebuild)
1717392039:  === (156 of 237) Merging (sci-mathematics/primesieve-11.0::/var/db/repos/gentoo/sci-mathematics/primesieve/primesieve-11.0.ebuild)
1717392040:  >>> AUTOCLEAN: sci-mathematics/primesieve:0
1717392041:  === (156 of 237) Post-Build Cleaning (sci-mathematics/primesieve-11.0::/var/db/repos/gentoo/sci-mathematics/primesieve/primesieve-11.0.ebuild)
1717392041:  ::: completed emerge (156 of 237) sci-mathematics/primesieve-11.0 to /
1717392041:  >>> emerge (157 of 237) dev-python/pyzmq-26.0.3 to /
1717392041:  === (157 of 237) Cleaning (dev-python/pyzmq-26.0.3::/var/db/repos/gentoo/dev-python/pyzmq/pyzmq-26.0.3.ebuild)
1717392042:  === (157 of 237) Compiling/Merging (dev-python/pyzmq-26.0.3::/var/db/repos/gentoo/dev-python/pyzmq/pyzmq-26.0.3.ebuild)
1717392063:  === (157 of 237) Merging (dev-python/pyzmq-26.0.3::/var/db/repos/gentoo/dev-python/pyzmq/pyzmq-26.0.3.ebuild)
1717392065:  >>> AUTOCLEAN: dev-python/pyzmq:0
1717392065:  === Unmerging... (dev-python/pyzmq-26.0.3)
1717392066:  >>> unmerge success: dev-python/pyzmq-26.0.3

Will investigate tomorrow.

[kiwifb]

equery d scikit-build - can wait tomorrow.

[strogdon]

# equery d scikit-build-core
 * These packages depend on scikit-build-core:
dev-python/pyzmq-26.0.3 (>=dev-python/scikit-build-core-0.9.4[python_targets_pypy3(-)?,python_targets_python3_10(-)?,python_targets_python3_11(-)?,python_targets_python3_12(-)?])

[strogdon]

Masking and removing dev-python/pyzmq-26.0.3, removing dev-python/scikit-build-core and updating world now allows sage-standard to build. Sorry for the delay. It was really late last evening.

[dimpase]

@minrk - it appears that using scikit in pyzmq build process creates problems for building SageMath

Can this be overcome with some kind of monkey-patching, or perhaps a fix at pyzmq?

[minrk]

Presumably the patch or fix belongs in either scikit-build-core or whatever in your build system is causing the build stages to be pickled, which doesn't seem like a typical situation or a safe assumption. That wouldn't be pyzmq, which I don't believe is in control when whatever is pickling things, especially since this will affect any packages using scikit-build-core, not just one. Unless I'm misreading the situation.

[kiwifb]

There are triggers that may be unusual. scikit-build-core interferes with sage building because sage happens to use multiprocessing as part of the process. I am pretty sure things would not fail if it did not.

[dimpase]

What happens if you use a saner way to build Sage, e.g. https://github.com/sagemath/sage/pull/36524 ?

[kiwifb]

I am seriously considering it, but this is a considerable PR and it is not that easy to just drop it in.

[kiwifb]

I must say that PR also has the issue of putting essential build system stuff in what is traditionally SAGE_ROOT. I think everything should be self contained in the src folder.

[kiwifb]

Why isn't this bit https://github.com/sagemath/sage/blob/e5f42fac70317c33655a3d3f882d732fdb635354/src/sage/misc/cython.py#L397 and after not work in this context? Does something like this need to be reproduced in sage_setup?

Also, the thing that really hurt is the lambda, if you replace it with some if statements, things works. On the other hand, the lambda reproduce stuff done in setuptools_rust (see https://github.com/scikit-build/scikit-build-core/issues/413) and I have no issues from it even so I have setuptools_rust installed.

[kiwifb]

I have very carefully looked at what setuptools-rust does compared to scikit-build-core and I have a patch that I may eventually submit to scikit upstream. It is very simple but it move things into key places where it does not do the damage it currently does to sage. But it needs to be more widely tested to make sure I got the right idea. I am putting it the repo for people to use in /etc/portage/patches/dev-python/scikit-build-core/ and experiment with it. https://github.com/cschwan/sage-on-gentoo/blob/master/tools/patches/scikit-build-core-entrypoint.patch

[kiwifb]

pyzmq seems to build fine with the patched scikit-build-core, but more things should be tried.

[strogdon]

Amazing, this does seem to work. The original issue was with Sage (maybe python) being unable to pickle a lambda in a package (scikit-build-core) which was not used by Sage. So why do the changes in a not needed package prevent Sage from trying to pickle the same lambda?

[dimpase]

I have very carefully looked at what setuptools-rust does compared to scikit-build-core and I have a patch that I may eventually submit to scikit upstream. It is very simple but it move things into key places where it does not do the damage it currently does to sage. But it needs to be more widely tested to make sure I got the right idea. I am putting it the repo for people to use in /etc/portage/patches/dev-python/scikit-build-core/ and experiment with it. https://github.com/cschwan/sage-on-gentoo/blob/master/tools/patches/scikit-build-core-entrypoint.patch

I'd submit it anyway - let them try it. Surely they have CI to test it better than us.

[kiwifb]

I think it needs a little bit of tidying up. The style of the code is not matching the re-plumbing I have made. I think some renaming following the kind of convention setuptools-rust has used is in order, but that's trivial.

The point of the matter is setuptools picks up on all extensions published in the entrypoint file in the egg-info directory. What scikit-build-core did was package their "cmake-extension" definition in the wrong entrypoint. It mostly works because of the internal implementation, but what I did is just move in a more appropriate entrypoint, which is not interfering with the building. And when you follow the setuptools-rust model, you end up simplify the code because you do not need to encapsulate everything in an object.

[kiwifb]

Submitted to scikit-build-core.

[dimpase]

seems they liked your patch, and merged it with changes. Good. Maybe we should explore how hard it is to build sagelib with cmake.

[kiwifb]

And they cut a new release including the stuff which means we will be able to move beyond the patch once that version is in the tree.

[kiwifb]

scikit-build-core-0.9.6 is in the tree and I have added to the list of keywords for sage-10.3 and sage-9999 so the issue should go away naturally now.

[kiwifb]

Considering this issue fixed now.