search

scikit-build / scikit-build-core

A next generation Python CMake adaptor and Python API for plugins
https://scikit-build-core.readthedocs.io
Apache License 2.0
232 stars 47 forks source link

Building with pyodide / emscripten on a mac produces bad wheel name (xxx-0.0.1-cp312-cp312-macosx_15_0_arm64.whl) #920

Open pthom opened 2 days ago

pthom commented 2 days ago

Hello

I'm trying to build a pyodide package using scikit-build-core, under macOS (ARM M1).

My issue is that the wheel that is produced is indeed a wasm wheel, but it is named "cp312-cp312-macosx_15_0_arm64.whl".

I know that this kind of issue is tricky because it is mixed in between pyodide and scikit-build-core. Anyhow I'd be interested in if a solution is possible.

Steps to reproduce

In order to reproduce the issue you can use the attached archive: daftsk.tgz

Simply extract it into pyodide/packages, then build it on a Mac with:

pyodide build-recipes daftsk --install

Here is a the archive content, which build a simple hello_world function that is bound to python:

daftsk
├── build
│   └── daftsk-0.0.1
│       ├── CMakeLists.txt
│       ├── dist
│       │   ├── daftsk-0.0.1-cp312-cp312-macosx_15_0_arm64.whl
│       │   └── unarchive
│       │       ├── daftsk
│       │       │   ├── __init__.py                                         # I unzipped the wheel here
│       │       │   └── _daftsk.cpython-312-darwin.so.     # and confirmed that this is a wasm binary module
│       │       └── daftsk-0.0.1.dist-info
│       │           ├── METADATA
│       │           ├── RECORD
│       │           └── WHEEL
│       ├── pyproject.toml
│       └── src
│           ├── daftsk
│           │   └── hello.cpp
│           └── python_bindings
│               ├── daftsk
│               │   └── __init__.py
│               └── module.cpp
├── build.log
├── meta.yaml
└── repo_daftsk
    ├── CMakeLists.txt
    ├── pyproject.toml
    └── src
        ├── daftsk
        │   └── hello.cpp
        └── python_bindings
            ├── daftsk
            │   └── __init__.py
            └── module.cpp

here is pyproject.toml

[build-system]
requires = ["scikit-build-core>=0.10.7", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "daftsk"
version = "0.0.1"
description="daftsk"
requires-python = ">=3.7"

[tool.scikit-build]
# wheel.expand-macos-universal-tags = true # No!!! Not for pyodide!!!
wheel.packages = ["src/python_bindings/daftsk"]
cmake.verbose = true
logging.level = "DEBUG"

and CMakeLists.txt:

cmake_minimum_required(VERSION 3.15...3.27)
project(litgen_template VERSION "0.0.1")
set(CMAKE_CXX_STANDARD 20)

# Build C++ library
add_library(daftsk STATIC src/daftsk/hello.cpp)

# Build bindings
find_package(Python 3.8 REQUIRED COMPONENTS Interpreter Development.Module)
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(_daftsk src/python_bindings/module.cpp)
target_link_libraries(_daftsk PRIVATE daftsk)
install(TARGETS _daftsk DESTINATION daftsk)

The wheel name is not ok, but its content is actually a wasm module: 

> file daftsk/_daftsk.cpython-312-darwin.so 
daftsk/_daftsk.cpython-312-darwin.so: WebAssembly (wasm) binary module version 0x1 (MVP)

Build logs

The build logs are attached in the archive. It ends with "*** Created daftsk-0.0.1-cp312-cp312-macosx_15_0_arm64.whl"

Below is a complete log (except for the final pyodide python exception), with logging.level = "DEBUG"

Running source pyodide_env in /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk                                                              
Creating isolated environment: venv+pip...
Installing packages in isolated environment:
- pybind11
- scikit-build-core>=0.10.7
Getting build dependencies for wheel...
WARNING: Use build.verbose instead of cmake.verbose for scikit-build-core >= 0.10
Building wheel...
WARNING: Use build.verbose instead of cmake.verbose for scikit-build-core >= 0.10
2024-10-01 14:23:09,264 - scikit_build_core - INFO - RUN: /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cmake -E capabilities
2024-10-01 14:23:09,401 - scikit_build_core - INFO - CMake version: 3.30.4
*** scikit-build-core 0.10.7 using CMake 3.30.4 (wheel)
2024-10-01 14:23:09,405 - scikit_build_core - DEBUG - MACOSX_DEPLOYMENT_TARGET not set, using 15.0
2024-10-01 14:23:09,405 - scikit_build_core - INFO - Build directory: /private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build
2024-10-01 14:23:09,406 - scikit_build_core - WARNING - No license files found, set wheel.license-files to [] to suppress this warning
*** Configuring CMake...
2024-10-01 14:23:09,407 - scikit_build_core - DEBUG - SITE_PACKAGES: /private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/lib/python3.12/site-packages
2024-10-01 14:23:09,596 - scikit_build_core - DEBUG - Default generator: Ninja
2024-10-01 14:23:09,597 - scikit_build_core - INFO - RUN: /opt/homebrew/bin/ninja --version
2024-10-01 14:23:09,600 - scikit_build_core - INFO - Ninja version: 1.11.0
2024-10-01 14:23:09,601 - scikit_build_core - DEBUG - CMAKE_GENERATOR: Using ninja: /opt/homebrew/bin/ninja
2024-10-01 14:23:09,602 - scikit_build_core - DEBUG - /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build/CMakeInit.txt:
  set(SKBUILD [===[2]===] CACHE STRING "" FORCE)
  set(SKBUILD_CORE_VERSION [===[0.10.7]===] CACHE STRING "" FORCE)
  set(SKBUILD_PROJECT_NAME [===[daftsk]===] CACHE STRING "" FORCE)
  set(SKBUILD_PROJECT_VERSION [===[0.0.1]===] CACHE STRING "" FORCE)
  set(SKBUILD_PROJECT_VERSION_FULL [===[0.0.1]===] CACHE STRING "" FORCE)
  set(PYTHON_EXECUTABLE [===[/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/bin/python]===] CACHE STRING "" FORCE)
  set(PYTHON_INCLUDE_DIR [===[/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12]===] CACHE PATH "" FORCE)
  set(PYTHON_LIBRARY [===[/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/lib/libpython3.12.a]===] CACHE PATH "" FORCE)
  set(Python_EXECUTABLE [===[/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/bin/python]===] CACHE STRING "" FORCE)
  set(Python_ROOT_DIR [===[/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq]===] CACHE STRING "" FORCE)
  set(Python_INCLUDE_DIR [===[/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12]===] CACHE PATH "" FORCE)
  set(Python_FIND_REGISTRY [===[NEVER]===] CACHE STRING "" FORCE)
  set(Python3_EXECUTABLE [===[/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/bin/python]===] CACHE STRING "" FORCE)
  set(Python3_ROOT_DIR [===[/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq]===] CACHE STRING "" FORCE)
  set(Python3_INCLUDE_DIR [===[/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12]===] CACHE PATH "" FORCE)
  set(Python3_FIND_REGISTRY [===[NEVER]===] CACHE STRING "" FORCE)
  set(SKBUILD_SOABI [===[cpython-312-wasm32-emscripten]===] CACHE STRING "" FORCE)
  set(SKBUILD_SABI_COMPONENT [===[]===] CACHE STRING "" FORCE)
  set(SKBUILD_PLATLIB_DIR [===[/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/platlib]===] CACHE PATH "" FORCE)
  set(SKBUILD_DATA_DIR [===[/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/data]===] CACHE PATH "" FORCE)
  set(SKBUILD_HEADERS_DIR [===[/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/headers]===] CACHE PATH "" FORCE)
  set(SKBUILD_SCRIPTS_DIR [===[/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/scripts]===] CACHE PATH "" FORCE)
  set(SKBUILD_NULL_DIR [===[/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/null]===] CACHE PATH "" FORCE)
  set(SKBUILD_METADATA_DIR [===[/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/metadata]===] CACHE PATH "" FORCE)
  set(SKBUILD_STATE [===[wheel]===] CACHE STRING "" FORCE)
  set(CMAKE_PREFIX_PATH [===[/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/lib/python3.12/site-packages]===] CACHE PATH "" FORCE)
  set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE "BOTH" CACHE PATH "")
2024-10-01 14:23:09,602 - scikit_build_core - DEBUG - RUNENV:
  AR='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ar'
  AUTOJUMP_ERROR_PATH='/Users/pascal/Library/autojump/errors.log'
  AUTOJUMP_SOURCED='1'
  BUILD_ENV_SCRIPTS_DIR='/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/bin'
  CARGO_BUILD_TARGET='wasm32-unknown-emscripten'
  CARGO_TARGET_WASM32_UNKNOWN_EMSCRIPTEN_LINKER='emcc'
  CC='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cc'
  CFLAGS_BASE='-O2 -g0 -fPIC'
  CMAKE_GENERATOR='Ninja'
  CMAKE_TOOLCHAIN_FILE='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages/pyodide_build/tools/cmake/Modules/Platform/Emscripten.cmake'
  COLORTERM='truecolor'
  COMMAND_MODE='unix2003'
  COMPILER_WRAPPER_DIR='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91'
  CPYTHONINSTALL='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1'
  CXX='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/c++'
  CXXFLAGS_BASE=''
  DISTDIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk/build/daftsk-0.0.1/dist'
  EMSDK='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk'
  EMSDK_NODE='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/node/18.20.3_64bit/bin/node'
  EMSDK_PYTHON='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/python/3.9.2_64bit/bin/python3'
  EM_CONFIG='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/.emscripten'
  EM_DIR='.'
  FC='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/gfortran'
  GCC='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/gcc'
  GEM_HOME='/Users/pascal/.gem/ruby/3.1.2'
  GEM_PATH='/Users/pascal/.gem/ruby/3.1.2:/Users/pascal/.rubies/ruby-3.1.2/lib/ruby/gems/3.1.0'
  GEM_ROOT='/Users/pascal/.rubies/ruby-3.1.2/lib/ruby/gems/3.1.0'
  GPG_TTY='/dev/ttys012'
  HOME='/Users/pascal'
  HOMEBREW_CELLAR='/opt/homebrew/Cellar'
  HOMEBREW_PREFIX='/opt/homebrew'
  HOMEBREW_REPOSITORY='/opt/homebrew'
  HOSTINSTALLDIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.artifacts'
  HOSTSITEPACKAGES='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.artifacts/lib/python3.12/site-packages'
  INFOPATH='/opt/homebrew/share/info:/opt/homebrew/share/info:'
  LANG='en_US.UTF-8'
  LD='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ld'
  LDFLAGS_BASE='-O2 -g0  -L/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/lib/ -s WASM_BIGINT'
  LESS='-R'
  LLD='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/lld'
  LOGNAME='pascal'
  LSCOLORS='Gxfxcxdxbxegedabagacad'
  LS_COLORS='di=1;36:ln=35:so=32:pi=33:ex=31:bd=34;46:cd=34;43:su=30;41:sg=30;46:tw=30;42:ow=30;43'
  LaunchInstanceID='DB67C00C-DE80-43A9-9F97-896F6F5BB384'
  MESON_CROSS_FILE='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages/pyodide_build/tools/emscripten.meson.cross'
  MallocNanoZone='0'
  NUMPY_LIB='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.artifacts/lib/python3.12/site-packages/numpy/'
  NVM_BIN='/Users/pascal/.nvm/versions/node/v22.2.0/bin'
  NVM_CD_FLAGS='-q'
  NVM_DIR='/Users/pascal/.nvm'
  NVM_INC='/Users/pascal/.nvm/versions/node/v22.2.0/include/node'
  ORIGINAL_XDG_CURRENT_DESKTOP='undefined'
  PAGER='less'
  PATH='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91:/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/bin:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/node_modules/.bin/:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/ccache/git-emscripten_64bit/bin:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/upstream/emscripten:/Users/pascal/.vscode/extensions/ms-python.python-2024.14.1-darwin-arm64/python_files/deactivate/zsh:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/bin:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/Users/pascal/.nvm/versions/node/v22.2.0/bin:/Users/pascal/.gem/ruby/3.1.2/bin:/Users/pascal/.rubies/ruby-3.1.2/lib/ruby/gems/3.1.0/bin:/Users/pascal/.rubies/ruby-3.1.2/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.9/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/go/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/pascal/.cargo/bin:/Users/pascal/.local/bin:/Users/pascal/bin:/Users/pascal/VulkanSDK/1.3.268.1/macOS/bin:/opt/homebrew/opt/fzf/bin:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.artifacts/bin/'
  PIP_CONSTRAINT='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/tools/constraints.txt'
  PKGDIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk'
  PKG_BUILD_DIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk/build/daftsk-0.0.1'
  PKG_CONFIG_LIBDIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.libs/lib/pkgconfig'
  PKG_VERSION='0.0.1'
  PLATFORM_TRIPLET='wasm32-emscripten'
  PWD='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk'
  PYMAJOR='3'
  PYMICRO='1'
  PYMINOR='12'
  PYO3_CROSS_INCLUDE_DIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12'
  PYO3_CROSS_LIB_DIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/sysconfigdata'
  PYO3_CROSS_PYTHON_VERSION='3.12'
  PYODIDE='1'
  PYODIDE_ABI_VERSION='2024_0'
  PYODIDE_EMSCRIPTEN_VERSION='3.1.58'
  PYODIDE_JOBS='1'
  PYODIDE_PACKAGE_ABI='1'
  PYODIDE_ROOT='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide'
  PYODIDE_VERSION='0.27.0.dev0'
  PYODIDE_ZIP_COMPRESSION_LEVEL='6'
  PYTHONINCLUDE='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12'
  PYTHONPATH='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/sysconfigdata'
  PYTHON_ARCHIVE_SHA256='d01ec6a33bc10009b09c17da95cc2759af5a580a7316b3a446eb4190e13f97b2'
  PYTHON_ARCHIVE_URL='https://www.python.org/ftp/python/3.12.1/Python-3.12.1.tgz'
  PYVERSION='3.12.1'
  PYWASMCROSS_ARGS='{"pkgname": "daftsk", "cflags": " -O2 -g0 -fPIC -I/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12", "cxxflags": " ", "ldflags": " -O2 -g0  -L/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/lib/ -s WASM_BIGINT -s SIDE_MODULE=1", "target_install_dir": "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1", "builddir": "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk", "exports": "pyinit", "PYTHONPATH": ["/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/bin", "/Library/Frameworks/Python.framework/Versions/3.12/lib/python312.zip", "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12", "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/lib-dynload", "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages", "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages/auditwheel_emscripten/third_party", "/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91", "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/sysconfigdata"], "orig__name__": "pyodide_build.pypabuild", "pythoninclude": "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12", "PATH": "/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/node_modules/.bin/:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/ccache/git-emscripten_64bit/bin:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/upstream/emscripten:/Users/pascal/.vscode/extensions/ms-python.python-2024.14.1-darwin-arm64/python_files/deactivate/zsh:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/bin:/opt/homebrew/opt/gnu-sed/libexec/gnubin:/Users/pascal/.nvm/versions/node/v22.2.0/bin:/Users/pascal/.gem/ruby/3.1.2/bin:/Users/pascal/.rubies/ruby-3.1.2/lib/ruby/gems/3.1.0/bin:/Users/pascal/.rubies/ruby-3.1.2/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Library/Frameworks/Python.framework/Versions/3.9/bin:/Library/Frameworks/Python.framework/Versions/3.8/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/Library/Frameworks/Python.framework/Versions/3.10/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/Library/TeX/texbin:/usr/local/share/dotnet:~/.dotnet/tools:/usr/local/go/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/pascal/.cargo/bin:/Users/pascal/.local/bin:/Users/pascal/bin:/Users/pascal/VulkanSDK/1.3.268.1/macOS/bin:/opt/homebrew/opt/fzf/bin:/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.artifacts/bin/"}'
  RANLIB='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ranlib'
  RUBYOPT=''
  RUBY_ENGINE='ruby'
  RUBY_ROOT='/Users/pascal/.rubies/ruby-3.1.2'
  RUBY_VERSION='3.1.2'
  RUSTFLAGS='-C link-arg=-sSIDE_MODULE=2 -C link-arg=-sWASM_BIGINT -Z link-native-libraries=no'
  RUST_TOOLCHAIN='nightly-2024-01-29'
  SECURITYSESSIONID='186ab'
  SHELL='/bin/zsh'
  SHLVL='2'
  SIDE_MODULE_CFLAGS='-O2 -g0 -fPIC -I/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12'
  SIDE_MODULE_CXXFLAGS=''
  SIDE_MODULE_LDFLAGS='-O2 -g0  -L/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/lib/ -s WASM_BIGINT -s SIDE_MODULE=1'
  SSH_AUTH_SOCK='/private/tmp/com.apple.launchd.C7OW1QP9Qa/Listeners'
  SSL_CERT_FILE='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/python/3.9.2_64bit/lib/python3.9/site-packages/certifi/cacert.pem'
  STDLIB_MODULE_CFLAGS='-O2 -g0 -fPIC -I/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12 -I Include/ -I. -IInclude/internal/'
  STRIP='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/strip'
  SYSCONFIGDATA_DIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/sysconfigdata/'
  SYSCONFIG_NAME='_sysconfigdata__emscripten_wasm32-emscripten'
  TARGETINSTALLDIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1'
  TERM='xterm-256color'
  TERM_PROGRAM='vscode'
  TERM_PROGRAM_VERSION='1.93.1'
  TMPDIR='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/'
  USER='pascal'
  USER_ZDOTDIR='/Users/pascal'
  VIRTUAL_ENV='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide'
  VIRTUAL_ENV_PROMPT='(venv_pyodide) '
  VK_ADD_LAYER_PATH='/Users/pascal/VulkanSDK/1.3.268.1/macOS/share/vulkan/explicit_layer.d'
  VK_DRIVER_FILES='/Users/pascal/VulkanSDK/1.3.268.1/macOS/share/vulkan/icd.d/MoltenVK_icd.json'
  VK_ICD_FILENAMES='/Users/pascal/VulkanSDK/1.3.268.1/macOS/share/vulkan/icd.d/MoltenVK_icd.json'
  VSCODE_INJECTION='1'
  VULKAN_SDK='/Users/pascal/VulkanSDK/1.3.268.1/macOS'
  WASM_LIBRARY_DIR='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/.libs'
  XPC_FLAGS='0x0'
  XPC_SERVICE_NAME='0'
  ZDOTDIR='/Users/pascal'
  ZSH='/Users/pascal/.oh-my-zsh'
  _='/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/bin/python3'
  _EMCC_CCACHE='1'
  _F2C_FIXES_WRAPPER=''
  _PYPROJECT_HOOKS_BUILD_BACKEND='scikit_build_core.build'
  _PYTHON_HOST_PLATFORM='emscripten_3_1_58_wasm32'
  _PYTHON_SYSCONFIGDATA_NAME='_sysconfigdata__emscripten_wasm32-emscripten'
  __CFBundleIdentifier='com.microsoft.VSCode'
  __CF_USER_TEXT_ENCODING='0x1F5:0x0:0x0'
2024-10-01 14:23:09,602 - scikit_build_core - INFO - RUN: /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cmake -S. -B/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build -DCMAKE_BUILD_TYPE:STRING=Release -C/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build/CMakeInit.txt -DCMAKE_INSTALL_PREFIX=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/platlib -DCMAKE_MAKE_PROGRAM=/opt/homebrew/bin/ninja
configure: cmake -DCMAKE_C_COMPILER=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cc -DCMAKE_CXX_COMPILER=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/c++ -DCMAKE_AR=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ar -DCMAKE_C_COMPILER_AR=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ar -DCMAKE_CXX_COMPILER_AR=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ar --fresh -S. -B/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build -DCMAKE_BUILD_TYPE:STRING=Release -C/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build/CMakeInit.txt -DCMAKE_INSTALL_PREFIX=/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/platlib -DCMAKE_MAKE_PROGRAM=/opt/homebrew/bin/ninja -DCMAKE_CROSSCOMPILING_EMULATOR=/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/node/18.20.3_64bit/bin/node
loading initial cache file /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build/CMakeInit.txt
-- Found Python: /private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/bin/python (found suitable version "3.12.0", minimum required is "3.8") found components: Interpreter Development.Module
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found pybind11: /private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/lib/python3.12/site-packages/pybind11/include (found version "2.13.6")
-- Configuring done (1.6s)
-- Generating done (0.0s)
-- Build files have been written to: /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build
*** Building project with Ninja...
2024-10-01 14:23:11,375 - scikit_build_core - DEBUG - RUNENV - changes since last run only:

2024-10-01 14:23:11,375 - scikit_build_core - INFO - RUN: /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cmake --build /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build -v
Change Dir: '/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build'

Run Build Command(s): /opt/homebrew/bin/ninja -v
[1/4] /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/c++   -O3 -DNDEBUG -std=gnu++20 -MD -MT CMakeFiles/daftsk.dir/src/daftsk/hello.cpp.o -MF CMakeFiles/daftsk.dir/src/daftsk/hello.cpp.o.d -o CMakeFiles/daftsk.dir/src/daftsk/hello.cpp.o -c /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk/build/daftsk-0.0.1/src/daftsk/hello.cpp
[2/4] : && /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages/cmake/data/bin/cmake -E rm -f libdaftsk.a && /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/ar qc libdaftsk.a  CMakeFiles/daftsk.dir/src/daftsk/hello.cpp.o && /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/emsdk/emsdk/upstream/emscripten/emranlib libdaftsk.a && /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages/cmake/data/bin/cmake -E touch libdaftsk.a && :
[3/4] /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/c++ -D_daftsk_EXPORTS -isystem /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12 -isystem /private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-7h706olq/lib/python3.12/site-packages/pybind11/include -O3 -DNDEBUG -std=gnu++20 -fPIC -fvisibility=hidden -fexceptions -flto -MD -MT CMakeFiles/_daftsk.dir/src/python_bindings/module.cpp.o -MF CMakeFiles/_daftsk.dir/src/python_bindings/module.cpp.o.d -o CMakeFiles/_daftsk.dir/src/python_bindings/module.cpp.o -c /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk/build/daftsk-0.0.1/src/python_bindings/module.cpp
[4/4] : && /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/c++ -fPIC -O3 -DNDEBUG  -fexceptions -flto -shared  -o _daftsk.cpython-312-darwin.so CMakeFiles/_daftsk.dir/src/python_bindings/module.cpp.o  libdaftsk.a && cd /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build && /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/strip /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build/_daftsk.cpython-312-darwin.so

*** Installing project into wheel...
2024-10-01 14:23:14,560 - scikit_build_core - DEBUG - RUNENV - changes since last run only:

2024-10-01 14:23:14,560 - scikit_build_core - INFO - RUN: /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cmake --install /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/build --prefix /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/platlib --strip
-- Install configuration: "Release"
-- Installing: /var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpgdrkcvvq/wheel/platlib/daftsk/_daftsk.cpython-312-darwin.so
*** Making wheel...
*** Created daftsk-0.0.1-cp312-cp312-macosx_15_0_arm64.whl
[2024-10-01 14:23:14] Failed building package daftsk in 7.0 seconds.                                                                                                       
Successfully built /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/packages/daftsk/build/daftsk-0.0.1/dist/daftsk-0.0.1-cp312-cp312-macosx_15_0_arm64.whl
╭─────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────╮
│ /Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/venv_pyodide/lib/python3.12/site-packages/pyodide_build/cli/build_recipes.py:117 in                     │
│ build_recipes_no_deps                                                                                                                                                   │
│                                                                                                                                                                         │
│   114 │   │   force_rebuild=force_rebuild,                                                                                                                              │
│   115 │   )                                                                                                                                                             │
│   116 │                                                                                                                                                                 │
│ ❱ 117 │   return build_recipes_no_deps_impl(packages, args, continue_)                                                                                                  │
│   118                                                                                                                                                                   │
│   119                                                                                                                                                                   │
│   120 def build_recipes_no_deps_impl(                                                                                                                                   │

=> then the build fail because of the name of the wheel

Many thanks in advance!

LecrisUT commented 2 days ago

What about if you run in cibuildwheel? Building locally with CMake it can easily escape and pick up local libraries, e.g. if there was a find_package(BLAS). I'm not familiar enough with such an environment to know if the following is correct:

CC='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cc'

Could you also print out all of the variables you find after running find_package(Python). Just to rule out that there is no system python that is leaking in there somehow.

This might also need support from upstream CMake to detect it (Python_INTERPRETER_ID) because iiuc it's Python_add_library that populates this field unless overwritten (Python_SOABI?).

pthom commented 1 day ago

I'm not familiar enough with such an environment to know if the following is correct:

I agree, this is quite a complex stack of technologies (python, C++, cmake, pyodide, scikit-build-core).

   CC='/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/tmpdf290p91/cc'

I guess this is part of the pyodide build system, which copies the emscripten compiler. Pyodide include a patched version of emsdk inside pyodide/emsdk/emsdk

Could you also print out all of the variables you find after running find_package(Python). Just to rule out that there is no system python that is leaking in there somehow.

Here are the relevant cmake variables after find_package(Python)

-- PYTHON_EXECUTABLE=/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-6b5o7tw1/bin/python
-- PYTHON_INCLUDE_DIR=/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12
-- PYTHON_LIBRARY=/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/lib/libpython3.12.a
-- Python3_EXECUTABLE=/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-6b5o7tw1/bin/python
-- Python3_FIND_REGISTRY=NEVER
-- Python3_INCLUDE_DIR=/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12
-- Python3_ROOT_DIR=/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-6b5o7tw1
-- Python_EXECUTABLE=/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-6b5o7tw1/bin/python
-- Python_FIND_REGISTRY=NEVER
-- Python_INCLUDE_DIR=/Users/pascal/dvp/OpenSource/ImGuiWork/_Bundle/_Pyodide/pyodide/cpython/installs/python-3.12.1/include/python3.12
-- Python_ROOT_DIR=/private/var/folders/w1/w77kvvl9613022ksfxj7v8xh0000gn/T/build-env-6b5o7tw1

This might also need support from upstream CMake to detect it (Python_INTERPRETER_ID) because iiuc it's Python_add_library that populates this field unless overwritten (Python_SOABI?).

This is a part where I must admit that I don't know.

What about if you run in cibuildwheel?

I have not explored using cibuildwheel together with pyodide. I think it might be very useful for the CI. However, my computer uses an ARM cpu, and I believe that cibuildwheel will handle pyodide builds with amd processors (which will be possible in CI, but not during development for me)

LecrisUT commented 1 day ago

I have not explored using cibuildwheel together with pyodide. I think it might be very useful for the CI. However, my computer uses an ARM cpu, and I believe that cibuildwheel will handle pyodide builds with amd processors (which will be possible in CI, but not during development for me)

Afaiu pyodide compiles to the browser's code (WebAssembly?) so it would be architecture and OS independent as long as you have the pyodide environment to run them locally. The scope of pyodide seems to be to make self-contained web applications using python, e.g. interactive jupyter notebooks or the whole holoviews widget ecosystem. @henryiii has more knowledge on this backend.

This might also need support from upstream CMake to detect it (Python_INTERPRETER_ID) because iiuc it's Python_add_library that populates this field unless overwritten (Python_SOABI?).

This is a part where I must admit that I don't know.

Try also to see what SKBUILD_SOABI gives you. See the warning section here on how to use it if it gives the correct tag. Other than that, everything looks fine, and my only guess is that CMake itself does not support the appropriate tag construction. Don't see any issues in CMake upstream so I'll open one after we gather some more information (primarily the difference between Python_SOABI given by FindPython by default and SKBUILD_SOABI which scikit-build-core constructs)

henryiii commented 1 day ago

This should work, and I know it does in cibuildwheel. pyodide build will rewrite binary with the correct SOABI for you, but I'm surprised that it doesn't fix the wheel tag. You are cross compiling, but these should work together. Quite a few pyodide package use scikit-bulid-core, like boost-histogram, iminuit, etc. I'll try to test this (much) later today (after teaching a class).

FYI, cibuildwheel is fine to use locally. Ignore the "ci" in the name. :) The binary you produce is in WASM, so the arch doesn't matter. That's one of the key features of WASM, it is the arch, and works on all system arches and platforms. Some tools even use this so they only have to ship one binary (though IIRC it's much easier for that use case to use WASI instead of Emscripten).

Also, pyodide's macOS building support is very limited and has several caveats. It's usually better to build in a docker container. pytest doesn't work, for example, on macOS, but python -m pytest does. Lots of weird stuff like that.