search

scikit-build / scikit-build-core

A next generation Python CMake adaptor and Python API for plugins
https://scikit-build-core.readthedocs.io
Apache License 2.0
240 stars 48 forks source link

Cross compiling fortran projects on cibuildwheel fails #505

Open zerothi opened 1 year ago

zerothi commented 1 year ago

I am trying to cross compile on Mac (for universal2) using github actions.

I am doing the most basic cibuildwheel github actions setup, so everything should be handled directly in cibuildwheel and associated variables.

Yet, when I compile my sources, see an excempt here, it is clear that the C sources gets compiled correctly (with -arch arm64 -arch x86_64) while fortran sources are missing the -arch arm64 -arch x86_64 flags).

I am a bit lost why these flags are not added here?

In my project I am also printing out the flags, from inside CMAKE, and there CMAKE_C_FLAGS and CMAKE_Fortran_FLAGS are both empty.

So my problem is:

  1. when is ARCHFLAGS added to the C-compiler, is it scikit-build-core, cmake, or cibuildwheel (in some intricate way)
  2. Why is this not added to the fortran compiler, and depending on 1, where should this be fixed? Thanks!
LecrisUT commented 1 year ago

Btw:

    gcc: warning: this compiler does not support Arm64 ('-arch' option ignored)

My understanding is that cmake needs a toolchain file in order to cross-compile to another architecture. I'm not sure how the cross-compilation is supposed to be controlled here.

You could also use qemu to emulate and build for linux arm. For mac it is a bit more tricky, but cirrusci provides macos arm builders and soon circleci will provide ones too. (Free ones I'm referring here)

zerothi commented 1 year ago

But how does cibuildwheel then pass the correct options to the C compiler? Since the flags are present for gcc, then it seems that CMake handles it fine, but I don't know how...

Note, this is to try and get the Mac wheels for pypi.

LecrisUT commented 1 year ago

You can do it through environment variables or specify in the pyproject.toml of cibuildwheel. https://cibuildwheel.readthedocs.io/en/stable/options/#config-settings

If there are some defaults baked in for cross-compiling, I am not sure, but @henryiii can clarify about both ends.

henryiii commented 1 year ago

this compiler does not support Arm64

Pretty sure that means you are dead in the water for cross-compiling to ARM with GCC. Sounds to me like they haven't implemented it, only AppleClang can cross compile, and no amount of CMake trickery help the compiler do something it can't. If LFortran or transpiling it to C isn't able to do it, I think you need to use GCC native on ARM. Cirrus can do it (with monthly limits), and GHA should get a public beta of ARM runners for macOS in about a month.

henryiii commented 1 year ago

What version of GCC?

LecrisUT commented 1 year ago

this compiler does not support Arm64

Pretty sure that means you are dead in the water for cross-compiling to ARM with GCC.

I thought that you need a gcc-arm64 compiler (e.g. gcc-c++-aarch64-linux-gnu package on fedora) that is picked up by toolchain file. Are those cross-compilers also made available on the main compiler through -arch flag? There is also the question which process has injected those options and why did it not get added to gfotran as well? Is cibuildwheel action/command assisting in that process?

zerothi commented 1 year ago

Ok, let's assume I'll fix the gcc problem.

I am more interested in the flags problem.

henryiii commented 1 year ago

cibuildwheels sets ARCHFLAGS. Scikit-build-core picks those up and sets CMAKE_OSX_ARCHITECTURES (CMake define) based on ARCHFLAGS.

zerothi commented 1 year ago

@henryiii thanks for this, that clarifies a few things.

Yet, I am puzzled why it is not added to the fortran flags?

I have now tried instead of doing enable_language, then moving Fortran into the project invocation (did not work).

I have also searched for this, and it seems that I can only find old problems related to the opposite problem.

I have also looked through the python_add_library in my local cmake shared location. I can't see anything that is out-of-place as it doesn't distinguish between source types. So I am a bit at a lost here. Should this be reported to cmake? And additionally it is hard for me to debug as I don't have access to a Mac laptop... :( Only through the github actions, which makes the workflow particularly bad... :(

Do you have any other ideas about the missing flags for the fortran compiler?

However, it seems to me that this isn't passed to fortran... :(

LecrisUT commented 1 year ago

I think this might simply be that there is no implementation for the gfortran (maybe fortran in general) for OSX cross-compilation on cmake. You can try your luck on their discourse channel or navigate the code in how OSX_ARCHITECTURES is used. It can either be in the compiler modules like /Modules/CMakeFortranCompiler.cmake.in or baked in the C++ code itself.

zerothi commented 1 year ago

Ok, i'll look around, and if problems still occur, I'll post this upstream.