Python: fix cross-compilation of extension modules

[FRidh]

Issue description

The changes in https://github.com/NixOS/nixpkgs/pull/53123 improve the cross-compilation situation of Python. It's now possible to produce functioning cross-compiled Python packages.

Neither the interpreter nor the packages will depend on the build machine. It does involve in both the interpreter and wrapPythonPrograms an ugly hack to rewrite shebangs from build Python to host Python. This is a general issue discussed in https://github.com/NixOS/nixpkgs/issues/33956. In this case I think we should somehow instruct distutils of our host Python.

Cross-compiling extension modules still fails. In the interpreter build we've nuked references to the compiler, and that seems to be an issue. When not cross-compiling, all goes fine though.

Steps to reproduce

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

[FRidh]

PYTHON_FOR_BUILD should supposedly always be set when cross-compiling. First, let's see whether we can set strictDeps = true; in buildPythonPackage and fix the outfall of that.

[Mic92]

cc @matthewbauer @Ericson2314

[FRidh]

Useful changeset https://github.com/SynoCommunity/spksrc/commit/6a071f8c24e2d067041fa4c362387668dfae9d46

[FRidh]

We need to keep _sysconfigdata*.py when cross-compiling because it's needed for extension modules. Probably best put it in a separate output so we don't have Python depending on dev outputs.

Using export PYTHON_FOR_BUILD=${getBin pythonForBuild.interpreter} seemed to break cross-compilation of Python itself.

[danbst]

Seems like we need a special Python for building extension modules (talking about cross-compiling numpy).

platform.machine() should return target platform

sysconfig.get_config_var('LDSHARED') should use target compiler.

[danbst]

Oh yeah, I've made numpy crosscompile, but with hacks. Given an overlay entry:

                python3 = super.python3.override {
                    packageOverrides = pythonself: pythonsuper: {
                        numpy = pythonsuper.numpy.overrideAttrs (old: {
                            _PYTHON_HOST_PLATFORM = pkgs.lib.removeSuffix "-" self.stdenv.cc.targetPrefix;
                            buildPython = self.buildPackages.python3;
                            patches = (old.patches or []) ++ [
                                (self.writeText "numpy-patch-cross-extensions.diff" ''
diff --git a/numpy/distutils/ccompiler.py b/numpy/distutils/ccompiler.py
index 14451fa..39dbf8f 100644
--- a/numpy/distutils/ccompiler.py
+++ b/numpy/distutils/ccompiler.py
@@ -140,7 +140,7 @@ def CCompiler_spawn(self, cmd, display=None):
             display = ' '.join(list(display))
     log.info(display)
     try:
-        subprocess.check_output(cmd)
+        subprocess.check_output([ (cmd[0] if cmd[0].startswith("${self.stdenv.cc.targetPrefix}") else "${self.stdenv.cc.targetPrefix}" + cmd[0]) ] + [ (arg.replace(os.environ['buildPython'], "${self.python3}")) for arg in cmd[1:]])
     except subprocess.CalledProcessError as exc:
         o = exc.output
         s = exc.returncode
diff --git a/numpy/random/setup.py b/numpy/random/setup.py
index a820d32..fa1d536 100644
--- a/numpy/random/setup.py
+++ b/numpy/random/setup.py
@@ -51,9 +51,9 @@ def configuration(parent_package=''', top_path=None):
         EXTRA_COMPILE_ARGS += ['-std=c99']
         INTEL_LIKE = any(arch in platform.machine()
                          for arch in ('x86', 'i686', 'i386', 'amd64'))
-        if INTEL_LIKE:
-            # Assumes GCC or GCC-like compiler
-            EXTRA_COMPILE_ARGS += ['-msse2']
+        #if INTEL_LIKE:
+        #    # Assumes GCC or GCC-like compiler
+        #    EXTRA_COMPILE_ARGS += ['-msse2']

     # Use legacy integer variable sizes
     LEGACY_DEFS = [('NP_RANDOM_LEGACY', '1')]
                                '')
                            ];
                        });
                    };
                    self = self.python3;
                };

it builds numpy without references to build python. @FRidh What is nice, your suggestion about nuked references wasn't an issue at all (e.g. it compiles even with refs nuked).

PS: I still haven't run this, so don't know if it works.

[FRidh]

What is nice, your suggestion about nuked references wasn't an issue at all (e.g. it compiles even with refs nuked).

That's convenient, though we of course need to check that for more packages.

A fix: buildPython = self.python.pythonForBuild Maybe we could add to the setup hook _PYTHON_HOST_PLATFORM if it is generic enough.

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[Ericson2314]

It sounds like the basic issue is that configure-script-like-things are not aware of cross compilation at all.

[FRidh]

https://github.com/NixOS/nixpkgs/pull/91178 was a generic fix. I am not aware of any further issues, but then again, I don't use cross-compilation myself.

[FRidh]

Splicing of Python packages in https://github.com/NixOS/nixpkgs/pull/104201.

[FRidh]

I consider this fixed with #98915 and #104135. Let's see until we find more needs to be done :smile:

Note pybind11 does not yet cross-compile, other than that all the packages I tried that have extension modules build.