search

PyO3 / maturin

Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages
https://maturin.rs
Apache License 2.0
3.7k stars 253 forks source link

Importing mixed Rust/Python project fails when structured as recommended #490

Closed ArniDagur closed 2 years ago

ArniDagur commented 3 years ago

The README of this project says the following:

Mixed rust/python projects

To create a mixed rust/python project, create a folder with your module name (i.e. lib.name in Cargo.toml) next to your Cargo.toml and add your python sources there:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   └── bar.py
├── Readme.md
└── src
    └── lib.rs

maturin will add the native extension as a module in your python folder. When using develop, maturin will copy the native library and for cffi also the glue code to your python folder. You should add those files to your gitignore.

With cffi you can do from .my_project import lib and then use lib.my_native_function, with pyo3/rust-cpython you can directly from .my_project import my_native_function.

Example layout with pyo3 after maturin develop:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   ├── bar.py
│   └── my_project.cpython-36m-x86_64-linux-gnu.so
├── Readme.md
└── src
    └── lib.rs

This is the structure I'm using for python-adblock. The project is written entirely in Rust, but I'm using a mixed Rust/Python project in order to include Python type stubs. I have an __init__.py which imports the neccesary symbols from the native module.

However, some of my users have reported that the imports inside __init__.py fail as follows:

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from .adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

or

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from adblock.adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

See https://github.com/ArniDagur/python-adblock/issues/17. As you can see, the result is the same whether I use from .adblock import ... or from adblock.adblock import .... I have only managed to reproduce this when the package is installed system wide (pip3 install .) and the current working directory is the root of my project (git repository).

Does anyone know what is going on? Am I doing something different from what is described in the README.md, or is the advice in the README.md wrong in some way?

omnivagant commented 3 years ago

I get the same kind of messages when trying to enable tests while upgrading the maturin aport itself https://gitlab.alpinelinux.org/omni/aports/-/jobs/368096#L691

messense commented 3 years ago

I get the same kind of messages when trying to enable tests while upgrading the maturin aport itself https://gitlab.alpinelinux.org/omni/aports/-/jobs/368096#L691

I'm not able to reproduce this with alpine:edge aarch64 container locally because alpine uses aarch64-alpine-linux-musl target which is not supported by maturin. Is the alpinelinux/alpine-gitlab-ci container doing something different than alpine:edge?

messense commented 3 years ago

I get the same kind of messages when trying to enable tests while upgrading the maturin aport itself https://gitlab.alpinelinux.org/omni/aports/-/jobs/368096#L691

I was able to reproduce with the target triple patch applied.

The problem is that maturin didn't take target_env into account in Target::get_shared_platform_tag(): https://github.com/PyO3/maturin/blob/ef37ef2a753557a17798a492d8b2f53920763fbd/src/target.rs#L171-L184

I was testing on aarch64 platform, it incorrectly named the .so file to aarch64-linux-gnu which should be aarch64-linux-musl according to imp.get_suffixes()

/io # ls /io/test-crates/venvs/pyo3-mixed-develop/lib/python3.9/site-packages/pyo3_mixed/
__init__.py                                  __pycache__                                  pyo3_mixed.cpython-39-aarch64-linux-gnu.so  python_module
/io # python
Python 3.9.4 (default, Apr  5 2021, 18:19:42)
[GCC 10.2.1 20210328] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import imp
<stdin>:1: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
>>> imp.get_suffixes()
[('.cpython-39-aarch64-linux-musl.so', 'rb', 3), ('.abi3.so', 'rb', 3), ('.so', 'rb', 3), ('.py', 'r', 1), ('.pyc', 'rb', 2)]

python3 -m sysconfig output

Platform: "linux-aarch64"
Python version: "3.9"
Current installation scheme: "posix_prefix"

Paths:
    data = "/usr"
    include = "/usr/include/python3.9"
    platinclude = "/usr/include/python3.9"
    platlib = "/usr/lib/python3.9/site-packages"
    platstdlib = "/usr/lib/python3.9"
    purelib = "/usr/lib/python3.9/site-packages"
    scripts = "/usr/bin"
    stdlib = "/usr/lib/python3.9"

Variables:
    ABIFLAGS = ""
    AC_APPLE_UNIVERSAL_BUILD = "0"
    AIX_BUILDDATE = "0"
    AIX_GENUINE_CPLUSPLUS = "0"
    ALT_SOABI = "0"
    ANDROID_API_LEVEL = "0"
    AR = "ar"
    ARFLAGS = "rcs"
    BASECFLAGS = "-Wno-unused-result -Wsign-compare"
    BASECPPFLAGS = ""
    BASEMODLIBS = ""
    BINDIR = "/usr/bin"
    BINLIBDEST = "/usr/lib/python3.9"
    BLDLIBRARY = "-L. -lpython3.9"
    BLDSHARED = "gcc -shared -fno-semantic-interposition  -fno-semantic-interposition"
    BUILDEXE = ""
    BUILDPYTHON = "python"
    BUILD_GNU_TYPE = "aarch64-alpine-linux-musl"
    BYTESTR_DEPS = "\"
    CC = "gcc"
    CCSHARED = "-fPIC"
    CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fno-semantic-interposition  -g -fno-semantic-interposition -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000"
    CFLAGSFORSHARED = "-fPIC"
    CFLAGS_ALIASING = ""
    CONFIGFILES = "configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
    CONFIGURE_CFLAGS = "-g -fno-semantic-interposition"
    CONFIGURE_CFLAGS_NODIST = "-flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden"
    CONFIGURE_CPPFLAGS = "-Os"
    CONFIGURE_LDFLAGS = "-fno-semantic-interposition"
    CONFIGURE_LDFLAGS_NODIST = "-flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g"
    CONFIG_ARGS = "'--build=aarch64-alpine-linux-musl' '--host=aarch64-alpine-linux-musl' '--prefix=/usr' '--enable-ipv6' '--enable-loadable-sqlite-extensions' '--enable-optimizations' '--enable-shared' '--with-lto' '--with-computed-gotos' '--with-dbmliborder=gdbm:ndbm' '--with-system-expat' '--with-system-ffi' '--with-system-libmpdec' '--without-ensurepip' '--with-bluetoothdir=/home/buildozer/aports/main/python3/src/bluez-5.55' 'build_alias=aarch64-alpine-linux-musl' 'host_alias=aarch64-alpine-linux-musl' 'CC=gcc' 'CFLAGS= -g -fno-semantic-interposition' 'LDFLAGS= -fno-semantic-interposition' 'CPPFLAGS=-Os'"
    CONFINCLUDEDIR = "/usr/include"
    CONFINCLUDEPY = "/usr/include/python3.9"
    COREPYTHONPATH = ""
    COVERAGE_INFO = "/home/buildozer/aports/main/python3/src/Python-3.9.4/coverage.info"
    COVERAGE_REPORT = "/home/buildozer/aports/main/python3/src/Python-3.9.4/lcov-report"
    COVERAGE_REPORT_OPTIONS = "--no-branch-coverage --title "CPython lcov report""
    CPPFLAGS = "-I. -I./Include -Os -Os"
    CXX = "g++"
    DESTDIRS = "/usr /usr/lib /usr/lib/python3.9 /usr/lib/python3.9/lib-dynload"
    DESTLIB = "/usr/lib/python3.9"
    DESTPATH = ""
    DESTSHARED = "/usr/lib/python3.9/lib-dynload"
    DFLAGS = "-Os -g"
    DIRMODE = "755"
    DIST = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in Include Lib Misc Ext-dummy"
    DISTDIRS = "Include Lib Misc Ext-dummy"
    DISTFILES = "README.rst ChangeLog configure configure.ac acconfig.h pyconfig.h.in Makefile.pre.in"
    DLINCLDIR = "."
    DLLLIBRARY = ""
    DOUBLE_IS_ARM_MIXED_ENDIAN_IEEE754 = "0"
    DOUBLE_IS_BIG_ENDIAN_IEEE754 = "0"
    DOUBLE_IS_LITTLE_ENDIAN_IEEE754 = "1"
    DTRACE = ""
    DTRACE_DEPS = "\"
    DTRACE_HEADERS = ""
    DTRACE_OBJS = ""
    DYNLOADFILE = "dynload_shlib.o"
    ENABLE_IPV6 = "1"
    ENSUREPIP = "no"
    EXE = ""
    EXEMODE = "755"
    EXPORTSFROM = ""
    EXPORTSYMS = ""
    EXTRATESTOPTS = ""
    EXT_SUFFIX = ".cpython-39-aarch64-linux-musl.so"
    FILEMODE = "644"
    FLOAT_WORDS_BIGENDIAN = "0"
    FLOCK_NEEDS_LIBBSD = "0"
    GETPGRP_HAVE_ARG = "0"
    GITBRANCH = ""
    GITTAG = ""
    GITVERSION = ""
    GNULD = "yes"
    HAVE_ACCEPT4 = "1"
    HAVE_ACOSH = "1"
    HAVE_ADDRINFO = "1"
    HAVE_ALARM = "1"
    HAVE_ALIGNED_REQUIRED = "0"
    HAVE_ALLOCA_H = "1"
    HAVE_ALTZONE = "0"
    HAVE_ASINH = "1"
    HAVE_ASM_TYPES_H = "1"
    HAVE_ATANH = "1"
    HAVE_BIND_TEXTDOMAIN_CODESET = "1"
    HAVE_BLUETOOTH_BLUETOOTH_H = "1"
    HAVE_BROKEN_MBSTOWCS = "0"
    HAVE_BROKEN_NICE = "0"
    HAVE_BROKEN_PIPE_BUF = "0"
    HAVE_BROKEN_POLL = "0"
    HAVE_BROKEN_POSIX_SEMAPHORES = "0"
    HAVE_BROKEN_PTHREAD_SIGMASK = "0"
    HAVE_BROKEN_SEM_GETVALUE = "0"
    HAVE_BROKEN_UNSETENV = "0"
    HAVE_BUILTIN_ATOMIC = "1"
    HAVE_CHFLAGS = "0"
    HAVE_CHOWN = "1"
    HAVE_CHROOT = "1"
    HAVE_CLOCK = "1"
    HAVE_CLOCK_GETRES = "1"
    HAVE_CLOCK_GETTIME = "1"
    HAVE_CLOCK_SETTIME = "1"
    HAVE_COMPUTED_GOTOS = "1"
    HAVE_CONFSTR = "1"
    HAVE_CONIO_H = "0"
    HAVE_COPYSIGN = "1"
    HAVE_COPY_FILE_RANGE = "1"
    HAVE_CRYPT_H = "1"
    HAVE_CRYPT_R = "1"
    HAVE_CTERMID = "1"
    HAVE_CTERMID_R = "0"
    HAVE_CURSES_FILTER = "1"
    HAVE_CURSES_H = "1"
    HAVE_CURSES_HAS_KEY = "1"
    HAVE_CURSES_IMMEDOK = "1"
    HAVE_CURSES_IS_PAD = "1"
    HAVE_CURSES_IS_TERM_RESIZED = "1"
    HAVE_CURSES_RESIZETERM = "1"
    HAVE_CURSES_RESIZE_TERM = "1"
    HAVE_CURSES_SYNCOK = "1"
    HAVE_CURSES_TYPEAHEAD = "1"
    HAVE_CURSES_USE_ENV = "1"
    HAVE_CURSES_WCHGAT = "1"
    HAVE_DECL_ISFINITE = "1"
    HAVE_DECL_ISINF = "1"
    HAVE_DECL_ISNAN = "1"
    HAVE_DECL_RTLD_DEEPBIND = "0"
    HAVE_DECL_RTLD_GLOBAL = "1"
    HAVE_DECL_RTLD_LAZY = "1"
    HAVE_DECL_RTLD_LOCAL = "1"
    HAVE_DECL_RTLD_MEMBER = "0"
    HAVE_DECL_RTLD_NODELETE = "1"
    HAVE_DECL_RTLD_NOLOAD = "1"
    HAVE_DECL_RTLD_NOW = "1"
    HAVE_DECL_TZNAME = "0"
    HAVE_DEVICE_MACROS = "1"
    HAVE_DEV_PTC = "0"
    HAVE_DEV_PTMX = "1"
    HAVE_DIRECT_H = "0"
    HAVE_DIRENT_D_TYPE = "1"
    HAVE_DIRENT_H = "1"
    HAVE_DIRFD = "1"
    HAVE_DLFCN_H = "1"
    HAVE_DLOPEN = "1"
    HAVE_DUP2 = "1"
    HAVE_DUP3 = "1"
    HAVE_DYLD_SHARED_CACHE_CONTAINS_PATH = "0"
    HAVE_DYNAMIC_LOADING = "1"
    HAVE_ENDIAN_H = "1"
    HAVE_EPOLL = "1"
    HAVE_EPOLL_CREATE1 = "1"
    HAVE_ERF = "1"
    HAVE_ERFC = "1"
    HAVE_ERRNO_H = "1"
    HAVE_EXECV = "1"
    HAVE_EXPLICIT_BZERO = "1"
    HAVE_EXPLICIT_MEMSET = "0"
    HAVE_EXPM1 = "1"
    HAVE_FACCESSAT = "1"
    HAVE_FCHDIR = "1"
    HAVE_FCHMOD = "1"
    HAVE_FCHMODAT = "1"
    HAVE_FCHOWN = "1"
    HAVE_FCHOWNAT = "1"
    HAVE_FCNTL_H = "1"
    HAVE_FDATASYNC = "1"
    HAVE_FDOPENDIR = "1"
    HAVE_FDWALK = "0"
    HAVE_FEXECVE = "1"
    HAVE_FINITE = "1"
    HAVE_FLOCK = "1"
    HAVE_FORK = "1"
    HAVE_FORKPTY = "1"
    HAVE_FPATHCONF = "1"
    HAVE_FSEEK64 = "0"
    HAVE_FSEEKO = "1"
    HAVE_FSTATAT = "1"
    HAVE_FSTATVFS = "1"
    HAVE_FSYNC = "1"
    HAVE_FTELL64 = "0"
    HAVE_FTELLO = "1"
    HAVE_FTIME = "1"
    HAVE_FTRUNCATE = "1"
    HAVE_FUTIMENS = "1"
    HAVE_FUTIMES = "1"
    HAVE_FUTIMESAT = "1"
    HAVE_GAI_STRERROR = "1"
    HAVE_GAMMA = "0"
    HAVE_GCC_ASM_FOR_MC68881 = "0"
    HAVE_GCC_ASM_FOR_X64 = "0"
    HAVE_GCC_ASM_FOR_X87 = "0"
    HAVE_GCC_UINT128_T = "1"
    HAVE_GETADDRINFO = "1"
    HAVE_GETC_UNLOCKED = "1"
    HAVE_GETENTROPY = "1"
    HAVE_GETGRGID_R = "1"
    HAVE_GETGRNAM_R = "1"
    HAVE_GETGROUPLIST = "1"
    HAVE_GETGROUPS = "1"
    HAVE_GETHOSTBYNAME = "0"
    HAVE_GETHOSTBYNAME_R = "1"
    HAVE_GETHOSTBYNAME_R_3_ARG = "0"
    HAVE_GETHOSTBYNAME_R_5_ARG = "0"
    HAVE_GETHOSTBYNAME_R_6_ARG = "1"
    HAVE_GETITIMER = "1"
    HAVE_GETLOADAVG = "1"
    HAVE_GETLOGIN = "1"
    HAVE_GETNAMEINFO = "1"
    HAVE_GETPAGESIZE = "1"
    HAVE_GETPEERNAME = "1"
    HAVE_GETPGID = "1"
    HAVE_GETPGRP = "1"
    HAVE_GETPID = "1"
    HAVE_GETPRIORITY = "1"
    HAVE_GETPWENT = "1"
    HAVE_GETPWNAM_R = "1"
    HAVE_GETPWUID_R = "1"
    HAVE_GETRANDOM = "1"
    HAVE_GETRANDOM_SYSCALL = "1"
    HAVE_GETRESGID = "1"
    HAVE_GETRESUID = "1"
    HAVE_GETSID = "1"
    HAVE_GETSPENT = "1"
    HAVE_GETSPNAM = "1"
    HAVE_GETWD = "0"
    HAVE_GLIBC_MEMMOVE_BUG = "0"
    HAVE_GRP_H = "1"
    HAVE_HSTRERROR = "1"
    HAVE_HTOLE64 = "1"
    HAVE_HYPOT = "1"
    HAVE_IEEEFP_H = "0"
    HAVE_IF_NAMEINDEX = "1"
    HAVE_INET_ATON = "1"
    HAVE_INET_PTON = "1"
    HAVE_INITGROUPS = "1"
    HAVE_INTTYPES_H = "1"
    HAVE_IO_H = "0"
    HAVE_IPA_PURE_CONST_BUG = "0"
    HAVE_KILL = "1"
    HAVE_KILLPG = "1"
    HAVE_KQUEUE = "0"
    HAVE_LANGINFO_H = "1"
    HAVE_LARGEFILE_SUPPORT = "0"
    HAVE_LCHFLAGS = "0"
    HAVE_LCHMOD = "0"
    HAVE_LCHOWN = "1"
    HAVE_LGAMMA = "1"
    HAVE_LIBDL = "1"
    HAVE_LIBDLD = "0"
    HAVE_LIBIEEE = "0"
    HAVE_LIBINTL_H = "0"
    HAVE_LIBREADLINE = "1"
    HAVE_LIBRESOLV = "0"
    HAVE_LIBSENDFILE = "0"
    HAVE_LIBUTIL_H = "0"
    HAVE_LINK = "1"
    HAVE_LINKAT = "1"
    HAVE_LINUX_CAN_BCM_H = "1"
    HAVE_LINUX_CAN_H = "1"
    HAVE_LINUX_CAN_J1939_H = "1"
    HAVE_LINUX_CAN_RAW_FD_FRAMES = "1"
    HAVE_LINUX_CAN_RAW_H = "1"
    HAVE_LINUX_CAN_RAW_JOIN_FILTERS = "1"
    HAVE_LINUX_MEMFD_H = "1"
    HAVE_LINUX_NETLINK_H = "1"
    HAVE_LINUX_QRTR_H = "1"
    HAVE_LINUX_RANDOM_H = "1"
    HAVE_LINUX_TIPC_H = "1"
    HAVE_LINUX_VM_SOCKETS_H = "1"
    HAVE_LINUX_WAIT_H = "1"
    HAVE_LOCKF = "1"
    HAVE_LOG1P = "1"
    HAVE_LOG2 = "1"
    HAVE_LONG_DOUBLE = "1"
    HAVE_LSTAT = "1"
    HAVE_LUTIMES = "1"
    HAVE_MADVISE = "1"
    HAVE_MAKEDEV = "1"
    HAVE_MBRTOWC = "1"
    HAVE_MEMFD_CREATE = "1"
    HAVE_MEMORY_H = "1"
    HAVE_MEMRCHR = "1"
    HAVE_MKDIRAT = "1"
    HAVE_MKFIFO = "1"
    HAVE_MKFIFOAT = "1"
    HAVE_MKNOD = "1"
    HAVE_MKNODAT = "1"
    HAVE_MKTIME = "1"
    HAVE_MMAP = "1"
    HAVE_MREMAP = "1"
    HAVE_NCURSES_H = "1"
    HAVE_NDIR_H = "0"
    HAVE_NETPACKET_PACKET_H = "1"
    HAVE_NET_IF_H = "1"
    HAVE_NICE = "1"
    HAVE_OPENAT = "1"
    HAVE_OPENPTY = "1"
    HAVE_PATHCONF = "1"
    HAVE_PAUSE = "1"
    HAVE_PIPE2 = "1"
    HAVE_PLOCK = "0"
    HAVE_POLL = "1"
    HAVE_POLL_H = "1"
    HAVE_POSIX_FADVISE = "1"
    HAVE_POSIX_FALLOCATE = "1"
    HAVE_POSIX_SPAWN = "1"
    HAVE_POSIX_SPAWNP = "1"
    HAVE_PREAD = "1"
    HAVE_PREADV = "1"
    HAVE_PREADV2 = "0"
    HAVE_PRLIMIT = "1"
    HAVE_PROCESS_H = "0"
    HAVE_PROTOTYPES = "1"
    HAVE_PTHREAD_CONDATTR_SETCLOCK = "1"
    HAVE_PTHREAD_DESTRUCTOR = "0"
    HAVE_PTHREAD_GETCPUCLOCKID = "1"
    HAVE_PTHREAD_H = "1"
    HAVE_PTHREAD_INIT = "0"
    HAVE_PTHREAD_KILL = "1"
    HAVE_PTHREAD_SIGMASK = "1"
    HAVE_PTY_H = "1"
    HAVE_PWRITE = "1"
    HAVE_PWRITEV = "1"
    HAVE_PWRITEV2 = "0"
    HAVE_READLINK = "1"
    HAVE_READLINKAT = "1"
    HAVE_READV = "1"
    HAVE_REALPATH = "1"
    HAVE_RENAMEAT = "1"
    HAVE_RL_APPEND_HISTORY = "1"
    HAVE_RL_CATCH_SIGNAL = "1"
    HAVE_RL_COMPLETION_APPEND_CHARACTER = "1"
    HAVE_RL_COMPLETION_DISPLAY_MATCHES_HOOK = "1"
    HAVE_RL_COMPLETION_MATCHES = "1"
    HAVE_RL_COMPLETION_SUPPRESS_APPEND = "1"
    HAVE_RL_PRE_INPUT_HOOK = "1"
    HAVE_RL_RESIZE_TERMINAL = "1"
    HAVE_ROUND = "1"
    HAVE_RTPSPAWN = "0"
    HAVE_SCHED_GET_PRIORITY_MAX = "1"
    HAVE_SCHED_H = "1"
    HAVE_SCHED_RR_GET_INTERVAL = "1"
    HAVE_SCHED_SETAFFINITY = "1"
    HAVE_SCHED_SETPARAM = "1"
    HAVE_SCHED_SETSCHEDULER = "1"
    HAVE_SEM_GETVALUE = "1"
    HAVE_SEM_OPEN = "1"
    HAVE_SEM_TIMEDWAIT = "1"
    HAVE_SEM_UNLINK = "1"
    HAVE_SENDFILE = "1"
    HAVE_SETEGID = "1"
    HAVE_SETEUID = "1"
    HAVE_SETGID = "1"
    HAVE_SETGROUPS = "1"
    HAVE_SETHOSTNAME = "1"
    HAVE_SETITIMER = "1"
    HAVE_SETLOCALE = "1"
    HAVE_SETPGID = "1"
    HAVE_SETPGRP = "1"
    HAVE_SETPRIORITY = "1"
    HAVE_SETREGID = "1"
    HAVE_SETRESGID = "1"
    HAVE_SETRESUID = "1"
    HAVE_SETREUID = "1"
    HAVE_SETSID = "1"
    HAVE_SETUID = "1"
    HAVE_SETVBUF = "1"
    HAVE_SHADOW_H = "1"
    HAVE_SHM_OPEN = "1"
    HAVE_SHM_UNLINK = "1"
    HAVE_SIGACTION = "1"
    HAVE_SIGALTSTACK = "1"
    HAVE_SIGFILLSET = "1"
    HAVE_SIGINFO_T_SI_BAND = "1"
    HAVE_SIGINTERRUPT = "1"
    HAVE_SIGNAL_H = "1"
    HAVE_SIGPENDING = "1"
    HAVE_SIGRELSE = "1"
    HAVE_SIGTIMEDWAIT = "1"
    HAVE_SIGWAIT = "1"
    HAVE_SIGWAITINFO = "1"
    HAVE_SNPRINTF = "1"
    HAVE_SOCKADDR_ALG = "1"
    HAVE_SOCKADDR_SA_LEN = "0"
    HAVE_SOCKADDR_STORAGE = "1"
    HAVE_SOCKETPAIR = "1"
    HAVE_SPAWN_H = "1"
    HAVE_SSIZE_T = "1"
    HAVE_STATVFS = "1"
    HAVE_STAT_TV_NSEC = "1"
    HAVE_STAT_TV_NSEC2 = "0"
    HAVE_STDARG_PROTOTYPES = "1"
    HAVE_STDINT_H = "1"
    HAVE_STDLIB_H = "1"
    HAVE_STD_ATOMIC = "1"
    HAVE_STRDUP = "1"
    HAVE_STRFTIME = "1"
    HAVE_STRINGS_H = "1"
    HAVE_STRING_H = "1"
    HAVE_STRLCPY = "1"
    HAVE_STROPTS_H = "1"
    HAVE_STRSIGNAL = "1"
    HAVE_STRUCT_PASSWD_PW_GECOS = "1"
    HAVE_STRUCT_PASSWD_PW_PASSWD = "1"
    HAVE_STRUCT_STAT_ST_BIRTHTIME = "0"
    HAVE_STRUCT_STAT_ST_BLKSIZE = "1"
    HAVE_STRUCT_STAT_ST_BLOCKS = "1"
    HAVE_STRUCT_STAT_ST_FLAGS = "0"
    HAVE_STRUCT_STAT_ST_GEN = "0"
    HAVE_STRUCT_STAT_ST_RDEV = "1"
    HAVE_STRUCT_TM_TM_ZONE = "1"
    HAVE_SYMLINK = "1"
    HAVE_SYMLINKAT = "1"
    HAVE_SYNC = "1"
    HAVE_SYSCONF = "1"
    HAVE_SYSEXITS_H = "1"
    HAVE_SYS_AUDIOIO_H = "0"
    HAVE_SYS_BSDTTY_H = "0"
    HAVE_SYS_DEVPOLL_H = "0"
    HAVE_SYS_DIR_H = "0"
    HAVE_SYS_ENDIAN_H = "0"
    HAVE_SYS_EPOLL_H = "1"
    HAVE_SYS_EVENT_H = "0"
    HAVE_SYS_FILE_H = "1"
    HAVE_SYS_IOCTL_H = "1"
    HAVE_SYS_KERN_CONTROL_H = "0"
    HAVE_SYS_LOADAVG_H = "0"
    HAVE_SYS_LOCK_H = "0"
    HAVE_SYS_MEMFD_H = "0"
    HAVE_SYS_MKDEV_H = "0"
    HAVE_SYS_MMAN_H = "1"
    HAVE_SYS_MODEM_H = "0"
    HAVE_SYS_NDIR_H = "0"
    HAVE_SYS_PARAM_H = "1"
    HAVE_SYS_POLL_H = "1"
    HAVE_SYS_RANDOM_H = "1"
    HAVE_SYS_RESOURCE_H = "1"
    HAVE_SYS_SELECT_H = "1"
    HAVE_SYS_SENDFILE_H = "1"
    HAVE_SYS_SOCKET_H = "1"
    HAVE_SYS_STATVFS_H = "1"
    HAVE_SYS_STAT_H = "1"
    HAVE_SYS_SYSCALL_H = "1"
    HAVE_SYS_SYSMACROS_H = "1"
    HAVE_SYS_SYS_DOMAIN_H = "0"
    HAVE_SYS_TERMIO_H = "0"
    HAVE_SYS_TIMES_H = "1"
    HAVE_SYS_TIME_H = "1"
    HAVE_SYS_TYPES_H = "1"
    HAVE_SYS_UIO_H = "1"
    HAVE_SYS_UN_H = "1"
    HAVE_SYS_UTSNAME_H = "1"
    HAVE_SYS_WAIT_H = "1"
    HAVE_SYS_XATTR_H = "1"
    HAVE_TCGETPGRP = "1"
    HAVE_TCSETPGRP = "1"
    HAVE_TEMPNAM = "1"
    HAVE_TERMIOS_H = "1"
    HAVE_TERM_H = "1"
    HAVE_TGAMMA = "1"
    HAVE_TIMEGM = "1"
    HAVE_TIMES = "1"
    HAVE_TMPFILE = "1"
    HAVE_TMPNAM = "1"
    HAVE_TMPNAM_R = "0"
    HAVE_TM_ZONE = "1"
    HAVE_TRUNCATE = "1"
    HAVE_TZNAME = "0"
    HAVE_UCS4_TCL = "0"
    HAVE_UNAME = "1"
    HAVE_UNISTD_H = "1"
    HAVE_UNLINKAT = "1"
    HAVE_USABLE_WCHAR_T = "1"
    HAVE_UTIL_H = "0"
    HAVE_UTIMENSAT = "1"
    HAVE_UTIMES = "1"
    HAVE_UTIME_H = "1"
    HAVE_UUID_CREATE = "0"
    HAVE_UUID_ENC_BE = "0"
    HAVE_UUID_GENERATE_TIME_SAFE = "0"
    HAVE_UUID_H = "0"
    HAVE_UUID_UUID_H = "0"
    HAVE_WAIT3 = "1"
    HAVE_WAIT4 = "1"
    HAVE_WAITID = "1"
    HAVE_WAITPID = "1"
    HAVE_WCHAR_H = "1"
    HAVE_WCSCOLL = "1"
    HAVE_WCSFTIME = "1"
    HAVE_WCSXFRM = "1"
    HAVE_WMEMCMP = "1"
    HAVE_WORKING_TZSET = "1"
    HAVE_WRITEV = "1"
    HAVE_X509_VERIFY_PARAM_SET1_HOST = "1"
    HAVE_ZLIB_COPY = "1"
    HAVE__GETPTY = "0"
    HOST_GNU_TYPE = "aarch64-alpine-linux-musl"
    INCLDIRSTOMAKE = "/usr/include /usr/include /usr/include/python3.9 /usr/include/python3.9"
    INCLUDEDIR = "/usr/include"
    INCLUDEPY = "/usr/include/python3.9"
    INSTALL = "/usr/bin/install -c"
    INSTALL_DATA = "/usr/bin/install -c -m 644"
    INSTALL_PROGRAM = "/usr/bin/install -c"
    INSTALL_SCRIPT = "/usr/bin/install -c"
    INSTALL_SHARED = "/usr/bin/install -c -m 755"
    INSTSONAME = "libpython3.9.so.1.0"
    IO_H = "Modules/_io/_iomodule.h"
    IO_OBJS = "\"
    LDCXXSHARED = "g++ -shared"
    LDFLAGS = "-fno-semantic-interposition  -fno-semantic-interposition"
    LDLIBRARY = "libpython3.9.so"
    LDLIBRARYDIR = ""
    LDSHARED = "gcc -shared -fno-semantic-interposition  -fno-semantic-interposition"
    LDVERSION = "3.9"
    LIBC = ""
    LIBDEST = "/usr/lib/python3.9"
    LIBDIR = "/usr/lib"
    LIBFFI_INCLUDEDIR = ""
    LIBM = "-lm"
    LIBOBJDIR = "Python/"
    LIBOBJS = ""
    LIBPC = "/usr/lib/pkgconfig"
    LIBPL = "/usr/lib/python3.9/config-3.9-aarch64-linux-musl"
    LIBPYTHON = ""
    LIBRARY = "libpython3.9.a"
    LIBRARY_OBJS = "\"
    LIBRARY_OBJS_OMIT_FROZEN = "\"
    LIBS = "-ldl  -lm"
    LIBSUBDIRS = "tkinter tkinter/test tkinter/test/test_tkinter \"
    LINKCC = "gcc"
    LINKFORSHARED = "-Xlinker -export-dynamic"
    LIPO_32BIT_FLAGS = ""
    LLVM_PROF_ERR = "no"
    LLVM_PROF_FILE = ""
    LLVM_PROF_MERGER = "true"
    LN = "ln"
    LOCALMODLIBS = ""
    MACHDEP = "linux"
    MACHDEP_OBJS = ""
    MACHDESTLIB = "/usr/lib/python3.9"
    MACOSX_DEPLOYMENT_TARGET = ""
    MAINCC = "gcc"
    MAJOR_IN_MKDEV = "0"
    MAJOR_IN_SYSMACROS = "1"
    MAKESETUP = "./Modules/makesetup"
    MANDIR = "/usr/share/man"
    MKDIR_P = "./install-sh -c -d"
    MODBUILT_NAMES = "posix  errno  pwd  _sre  _codecs  _weakref  _functools  _operator  _collections  _abc  itertools  atexit  _signal  _stat  time  _thread  _locale  _io  faulthandler  _tracemalloc  _peg_parser  _symtable  xxsubtype"
    MODDISABLED_NAMES = ""
    MODLIBS = ""
    MODOBJS = "Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o  Modules/_peg_parser.o  Modules/symtablemodule.o  Modules/xxsubtype.o"
    MODULE_OBJS = "\"
    MULTIARCH = "aarch64-linux-musl"
    MULTIARCH_CPPFLAGS = "-DMULTIARCH=\"aarch64-linux-musl\""
    MVWDELCH_IS_EXPRESSION = "1"
    NO_AS_NEEDED = "-Wl,--no-as-needed"
    OBJECT_OBJS = "\"
    OPENSSL_INCLUDES = ""
    OPENSSL_LDFLAGS = ""
    OPENSSL_LIBS = "-lssl -lcrypto"
    OPT = "-DNDEBUG -g -fwrapv -O3 -Wall"
    OTHER_LIBTOOL_OPT = ""
    PACKAGE_BUGREPORT = "0"
    PACKAGE_NAME = "0"
    PACKAGE_STRING = "0"
    PACKAGE_TARNAME = "0"
    PACKAGE_URL = "0"
    PACKAGE_VERSION = "0"
    PARSER_HEADERS = "\"
    PARSER_OBJS = "\ \ Parser/myreadline.o Parser/parsetok.o Parser/tokenizer.o"
    PEGEN_HEADERS = "\"
    PEGEN_OBJS = "\"
    PGO_PROF_GEN_FLAG = "-fprofile-generate"
    PGO_PROF_USE_FLAG = "-fprofile-use -fprofile-correction"
    PLATLIBDIR = "lib"
    POBJS = "\"
    POSIX_SEMAPHORES_NOT_ENABLED = "0"
    PROFILE_TASK = "-m test --pgo"
    PTHREAD_KEY_T_IS_COMPATIBLE_WITH_INT = "1"
    PTHREAD_SYSTEM_SCHED_SUPPORTED = "1"
    PURIFY = ""
    PY3LIBRARY = "libpython3.so"
    PYLONG_BITS_IN_DIGIT = "0"
    PYTHON = "python"
    PYTHONFRAMEWORK = ""
    PYTHONFRAMEWORKDIR = "no-framework"
    PYTHONFRAMEWORKINSTALLDIR = ""
    PYTHONFRAMEWORKPREFIX = ""
    PYTHONPATH = ""
    PYTHON_FOR_BUILD = "./python -E"
    PYTHON_HEADERS = "\"
    PYTHON_OBJS = "\"
    PY_BUILTIN_HASHLIB_HASHES = ""md5,sha1,sha256,sha512,sha3,blake2""
    PY_BUILTIN_MODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fno-semantic-interposition  -g -fno-semantic-interposition -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000 -flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-use -fprofile-correction -I./Include/internal -I. -I./Include -Os -Os -fPIC -DPy_BUILD_CORE_BUILTIN"
    PY_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fno-semantic-interposition  -g -fno-semantic-interposition -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000"
    PY_CFLAGS_NODIST = "-flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-use -fprofile-correction -I./Include/internal"
    PY_COERCE_C_LOCALE = "1"
    PY_CORE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fno-semantic-interposition  -g -fno-semantic-interposition -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000 -flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-use -fprofile-correction -I./Include/internal -I. -I./Include -Os -Os -fPIC -DPy_BUILD_CORE"
    PY_CORE_LDFLAGS = "-fno-semantic-interposition  -fno-semantic-interposition -flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g"
    PY_CPPFLAGS = "-I. -I./Include -Os -Os"
    PY_FORMAT_SIZE_T = ""z""
    PY_LDFLAGS = "-fno-semantic-interposition  -fno-semantic-interposition"
    PY_LDFLAGS_NODIST = "-flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g"
    PY_SSL_DEFAULT_CIPHERS = "1"
    PY_SSL_DEFAULT_CIPHER_STRING = "0"
    PY_STDMODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fno-semantic-interposition  -g -fno-semantic-interposition -g -fno-semantic-interposition -DTHREAD_STACK_SIZE=0x100000 -flto=4 -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -fprofile-use -fprofile-correction -I./Include/internal -I. -I./Include -Os -Os -fPIC"
    Py_DEBUG = "0"
    Py_ENABLE_SHARED = "1"
    Py_HASH_ALGORITHM = "0"
    Py_TRACE_REFS = "0"
    QUICKTESTOPTS = "-x test_subprocess test_io test_lib2to3 \"
    READELF = "readelf"
    RESSRCDIR = "Mac/Resources/framework"
    RETSIGTYPE = "void"
    RUNSHARED = "LD_LIBRARY_PATH=/home/buildozer/aports/main/python3/src/Python-3.9.4"
    SCRIPTDIR = "/usr/lib"
    SETPGRP_HAVE_ARG = "0"
    SGI_ABI = "@SGI_ABI@"
    SHELL = "/bin/sh"
    SHLIBS = "-ldl  -lm"
    SHLIB_SUFFIX = ".so"
    SHM_NEEDS_LIBRT = "0"
    SIGNED_RIGHT_SHIFT_ZERO_FILLS = "0"
    SITEPATH = ""
    SIZEOF_DOUBLE = "8"
    SIZEOF_FLOAT = "4"
    SIZEOF_FPOS_T = "16"
    SIZEOF_INT = "4"
    SIZEOF_LONG = "8"
    SIZEOF_LONG_DOUBLE = "16"
    SIZEOF_LONG_LONG = "8"
    SIZEOF_OFF_T = "8"
    SIZEOF_PID_T = "4"
    SIZEOF_PTHREAD_KEY_T = "4"
    SIZEOF_PTHREAD_T = "8"
    SIZEOF_SHORT = "2"
    SIZEOF_SIZE_T = "8"
    SIZEOF_TIME_T = "8"
    SIZEOF_UINTPTR_T = "8"
    SIZEOF_VOID_P = "8"
    SIZEOF_WCHAR_T = "4"
    SIZEOF__BOOL = "1"
    SO = ".cpython-39-aarch64-linux-musl.so"
    SOABI = "cpython-39-aarch64-linux-musl"
    SRCDIRS = "Parser Parser/pegen Objects Python Modules Modules/_io Programs"
    SRC_GDB_HOOKS = "./Tools/gdb/libpython.py"
    STDC_HEADERS = "1"
    STRICT_SYSV_CURSES = "/* Don't use ncurses extensions */"
    STRIPFLAG = "-s"
    SUBDIRS = ""
    SUBDIRSTOO = "Include Lib Misc"
    SYSLIBS = "-lm"
    SYS_SELECT_WITH_SYS_TIME = "1"
    TCLTK_INCLUDES = ""
    TCLTK_LIBS = ""
    TESTOPTS = ""
    TESTPATH = ""
    TESTPYTHON = "LD_LIBRARY_PATH=/home/buildozer/aports/main/python3/src/Python-3.9.4 ./python"
    TESTPYTHONOPTS = ""
    TESTRUNNER = "LD_LIBRARY_PATH=/home/buildozer/aports/main/python3/src/Python-3.9.4 ./python ./Tools/scripts/run_tests.py"
    TESTTIMEOUT = "1200"
    TIMEMODULE_LIB = "0"
    TIME_WITH_SYS_TIME = "1"
    TM_IN_SYS_TIME = "0"
    TZPATH = "/usr/share/zoneinfo:/usr/lib/zoneinfo:/usr/share/lib/zoneinfo:/etc/zoneinfo"
    UNICODE_DEPS = "\"
    UNIVERSALSDK = ""
    UPDATE_FILE = "python3 ./Tools/scripts/update_file.py"
    USE_COMPUTED_GOTOS = "1"
    VERSION = "3.9"
    WINDOW_HAS_FLAGS = "1"
    WITH_DECIMAL_CONTEXTVAR = "1"
    WITH_DOC_STRINGS = "1"
    WITH_DTRACE = "0"
    WITH_DYLD = "0"
    WITH_LIBINTL = "0"
    WITH_NEXT_FRAMEWORK = "0"
    WITH_PYMALLOC = "1"
    WITH_VALGRIND = "0"
    X87_DOUBLE_ROUNDING = "0"
    XMLLIBSUBDIRS = "xml xml/dom xml/etree xml/parsers xml/sax"
    abiflags = ""
    abs_builddir = "/home/buildozer/aports/main/python3/src/Python-3.9.4"
    abs_srcdir = "/home/buildozer/aports/main/python3/src/Python-3.9.4"
    base = "/usr"
    datarootdir = "/usr/share"
    exec_prefix = "/usr"
    installed_base = "/usr"
    installed_platbase = "/usr"
    platbase = "/usr"
    platlibdir = "lib"
    prefix = "/usr"
    projectbase = "/usr/bin"
    py_version = "3.9.4"
    py_version_nodot = "39"
    py_version_short = "3.9"
    srcdir = "/usr/lib/python3.9/config-3.9-aarch64-linux-musl"
    userbase = "/root/.local"

You can patch src/target.rs to make it work:

diff --git a/src/target.rs b/src/target.rs
index b033cbe..34817a6 100644
--- a/src/target.rs
+++ b/src/target.rs
@@ -171,12 +171,24 @@ impl Target {
     pub fn get_shared_platform_tag(&self) -> &'static str {
         match (&self.os, &self.arch) {
             (Os::FreeBsd, _) => "", // according imp.get_suffixes(), there are no such
-            (Os::Linux, Arch::Aarch64) => "aarch64-linux-gnu", // aka armv8-linux-gnueabihf
+            (Os::Linux, Arch::Aarch64) => {
+                if self.is_musl_target() {
+                    "aarch64-linux-musl"
+                } else {
+                    "aarch64-linux-gnu"
+                }
+            }
             (Os::Linux, Arch::Armv7L) => "arm-linux-gnueabihf",
             (Os::Linux, Arch::Powerpc64) => "powerpc64-linux-gnu",
             (Os::Linux, Arch::Powerpc64Le) => "powerpc64le-linux-gnu",
             (Os::Linux, Arch::X86) => "i386-linux-gnu", // not i686
-            (Os::Linux, Arch::X86_64) => "x86_64-linux-gnu",
+            (Os::Linux, Arch::X86_64) => {
+                if self.is_musl_target() {
+                    "x86_64-linux-musl"
+                } else {
+                    "x86_64-linux-gnu"
+                }
+            }
             (Os::Linux, Arch::S390X) => "s390x-linux-gnu",
             (Os::Macos, Arch::X86_64) => "darwin",
             (Os::Macos, Arch::Aarch64) => "darwin",
@@ -366,6 +378,7 @@ fn get_host_target() -> Result<String> {
                 output
             )
         })?
+        .replace("alpine", "unknown")
         .to_string();
     Ok(host)
 }
omnivagant commented 3 years ago

Thanks! I'll try!

omnivagant commented 3 years ago

I added a target-musl.patch with your suggestions and similar for the archs we support (or at least want to support, ppc64le is limited by the ring crate and we currently don't have rust at all for s390x), with success for x86_64 and aarch64!

But I'm still having issues with x86 (as well armv7, but that currently seem to be of another character).

messense commented 3 years ago

@omnivagant Can you try this patch instead? https://github.com/messense/maturin/commit/5ad4f178e5176891adedfdb7d6b7da351faaabdc

I was able to get tests passing with it on x86_64 macOS using a alpine:edge linux/386 docker container.

omnivagant commented 3 years ago

Yes, I've replaced my previous target-musl.patch with your suggested patch as use-sysconfig-ext_suffix.patch and this seem to work for aarch64, x86 and x86_64, thanks!

I still have issues with armv7 (and probably armhf, but that is not in our CI so I cannot check) but that seem to be a separate issue to this.

messense commented 3 years ago

I'm having trouble testing in armv7 alpine:edge docker container so I can't look into it.

/io # cargo build
    Updating crates.io index
warning: spurious network error (2 tries remaining): failed to mmap. Could not write data: Out of memory; class=Os (2)
warning: spurious network error (1 tries remaining): failed to mmap. Could not write data: Out of memory; class=Os (2)
error: failed to get `anyhow` as a dependency of package `maturin v0.10.2 (/io)`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to mmap. Could not write data: Out of memory; class=Os (2)
messense commented 3 years ago

ERROR: cffi_mixed-0.1.0-py3-none-linux_armv7l.whl is not a supported wheel on this platform. ERROR: cffi_pure-0.1.0-py3-none-linux_armv7l.whl is not a supported wheel on this platform. ERROR: pyo3_mixed-2.1.3-cp39-cp39-linux_armv7l.whl is not a supported wheel on this platform.

The error message is a little confusing, pip should support them according to pip debug --verbose


# pip debug --verbose
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 20.3.4 from /usr/lib/python3.9/site-packages/pip (python 3.9)
sys.version: 3.9.4 (default, Apr  5 2021, 18:19:42)
[GCC 10.2.1 20210328]
sys.executable: /usr/bin/python3
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /etc/ssl/certs/ca-certificates.crt
pip._vendor.DEBUNDLED: False
vendored library versions:
  appdirs==1.4.4
  CacheControl==0.12.6
  colorama==0.4.4
  contextlib2==0.6.0.post1 (Unable to locate actual module version, using vendor.txt specified version)
  distlib==0.3.1
  distro==1.5.0 (Unable to locate actual module version, using vendor.txt specified version)
  html5lib==1.1
  ipaddress==1.0.23
  msgpack==1.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  packaging==20.8
  pep517==0.9.1
  progress==1.5
  pyparsing==2.4.7
  requests==2.25.0
  certifi==2020.11.08
  chardet==3.0.4
  idna==2.10
  urllib3==1.26.2
  resolvelib==0.5.4
  retrying==1.3.3 (Unable to locate actual module version, using vendor.txt specified version)
  setuptools==44.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.15.0
  toml==0.10.2
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 33
  cp39-cp39-linux_armv7l
  cp39-abi3-linux_armv7l
  cp39-none-linux_armv7l
  cp38-abi3-linux_armv7l
  cp37-abi3-linux_armv7l
  cp36-abi3-linux_armv7l
  cp35-abi3-linux_armv7l
  cp34-abi3-linux_armv7l
  cp33-abi3-linux_armv7l
  cp32-abi3-linux_armv7l
  py39-none-linux_armv7l
  py3-none-linux_armv7l
  py38-none-linux_armv7l
  py37-none-linux_armv7l
  py36-none-linux_armv7l
  py35-none-linux_armv7l
  py34-none-linux_armv7l
  py33-none-linux_armv7l
  py32-none-linux_armv7l
  py31-none-linux_armv7l
  py30-none-linux_armv7l
  cp39-none-any
  py39-none-any
  py3-none-any
  py38-none-any
  py37-none-any
  py36-none-any
  py35-none-any
  py34-none-any
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any
konstin commented 3 years ago

I haven't followed this thread closely (thanks @messense for handling this), but it would be great if those patches could eventually be upstreamed in a way that maturin works on both glibc based linux and on alpine.

Re powerpc: It works, you just need to deactivate default features.

omnivagant commented 3 years ago

Hi!

WRT ppc64le, cool! I'll give it a try!

omnivagant commented 3 years ago

I tried to enable for ppc64le and got

error: failed to run custom build command for `ring v0.16.20`

Caused by:
  process didn't exit successfully: `/builds/omni/aports/community/maturin/src/maturin-0.10.2/target/release/build/ring-e05f73076e6d300b/build-script-build` (exit code: 101)
  --- stderr
  thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /home/buildozer/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.20/build.rs:358:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

But I feel more and more that I've hijacked this issue with things that aren't part of the original one. :upside_down_face: The ring crate has an issue for this.

messense commented 3 years ago

I've done an armv7 test on Raspberry Pi 3B+ using a alpine:edge armv7 docker container, the current set of patches works great and cargo test passed. So I'm not sure what's going on with alpine's CI since we don't know what hardware it is running on, is it running with virtualization or emulation.

messense commented 3 years ago

I'm having trouble testing in armv7 alpine:edge docker container so I can't look into it.

/io # cargo build
    Updating crates.io index
warning: spurious network error (2 tries remaining): failed to mmap. Could not write data: Out of memory; class=Os (2)
warning: spurious network error (1 tries remaining): failed to mmap. Could not write data: Out of memory; class=Os (2)
error: failed to get `anyhow` as a dependency of package `maturin v0.10.2 (/io)`

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  failed to mmap. Could not write data: Out of memory; class=Os (2)

So I've workaround the issue by manually clone a crates.io-index in the container:

cd ~/.cargo/registry/index
git clone --bare https://github.com/rust-lang/crates.io-index.git github.com-1285ae84e5963aae

and cargo test passed without any issue. 😂

omnivagant commented 3 years ago

Alpine CI builders and package builders are hardware, not sure on the specifics but some server grade. Recently an armhf builder was added to CI.

omnivagant commented 3 years ago

I haven't dug into it (again) yet but recently enabled armv7 and armhf in an MR over at our gitlab, here are the logs for the jobs failing on these two archs:

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391373

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391374

messense commented 3 years ago

I haven't dug into it (again) yet but recently enabled armv7 and armhf in an MR over at our gitlab, here are the logs for the jobs failing on these two archs:

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391373

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391374

Try pip debug --verbose

omnivagant commented 3 years ago

pip isn't used, but I've added python3 -m sysconfig for these archs.

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391397

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391398

messense commented 3 years ago

@omnivagant pip is used in maturin's tests: https://github.com/PyO3/maturin/blob/5b7efcc7683ee233388e8d6f6e52347ebc85a588/tests/common/integration.rs#L85-L105

messense commented 3 years ago

pip isn't used, but I've added python3 -m sysconfig for these archs.

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391397

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391398

armv7: Platform: "linux-armv8l" armhf: Platform: "linux-armv8l"

Interesting. I think we are expecting Platform: "linux-armv7l"

I wonder what's is Rust target triple on these platforms.

See also https://github.com/randombit/botan/issues/1543#issuecomment-382186505

messense commented 3 years ago

@omnivagant Can you try this patch? https://github.com/messense/maturin/commit/e518086b0927af1dd64b5639d8b60084b792ffe5

omnivagant commented 3 years ago

Oh, I didn't think pip was available since it isn't in makedepends or checkdepends or as a dependency of anything there (I think).

Builds are running with your patch now.

omnivagant commented 3 years ago

But failed. I'll try to dive into this soon.

messense commented 3 years ago

But failed. I'll try to dive into this soon.

It should be easy to fix. Just need to pip install cffi before installing generated wheel.

messense commented 3 years ago

But failed. I'll try to dive into this soon.

It should be easy to fix. Just need to pip install cffi before installing generated wheel.

@omnivagant New patch: https://github.com/messense/maturin/commit/c299b71f6f3893aa7035b0a63978bbd074ec94cb Please give it a try. I've opened https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/21694

omnivagant commented 3 years ago

Cool! I've been too occupied with other things lately, sorry.

stusmall commented 2 years ago

I'm running into this same issue, but I might have a shorter path to reproducing it. I'm including as much detail as possible because I might be doing something weird in here. I'm not an experienced python dev and I'm kinda winging it as I go.

  1. Check out maturin @ revision d531e49e6b4d106ac2fa5bac2f4a282570da3731 (head of main when I pulled)
  2. Install with cargo install -f --path .
  3. Goto the cffi-mixed test crate with cd test-crates/cffi-mixed
  4. Create and activate a venv. python3 -m venv venv; source venv/bin/activate
  5. Install tox pip install tox
  6. Run tox tox --skip-missing-interpreters (I'm just using the system interpreter to simply things): 
    
(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
.package create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.package
.package installdeps: maturin>=0.12,<0.13
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38
py38 installdeps: pytest
py38 inst: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='1058416854'
py38 run-test: commands[0] | pytest
================================= test session starts ==================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 0 items / 1 error

======================================== ERRORS ======================================== __ ERROR collecting test-crates/cffi-mixed/test_cffi_mixed.py __ ImportError while importing test module '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/test_cffi_mixed.py'. Hint: make sure your test modules/packages have valid Python names. Traceback: ../../../../.pyenv/versions/3.8.12/lib/python3.8/importlib/init.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) test_cffi_mixed.py:3: in import cffi_mixed cffi_mixed/init.py:1: in from .cffi_mixed import ffi, lib E ModuleNotFoundError: No module named 'cffi_mixed.cffi_mixed' =============================== short test summary info ================================ ERROR test_cffi_mixed.py !!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!! =================================== 1 error in 0.08s =================================== ERROR: InvocationError for command /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38/bin/pytest (exited with code 2) ___ summary ____ SKIPPED: py36: InterpreterNotFound: python3.6 SKIPPED: py37: InterpreterNotFound: python3.7 ERROR: py38: commands failed


For some extra version info just in case it helps for reproducing the issue:

(venv) ➜ cffi-mixed git:(main) python3 --version Python 3.8.12 (venv) ➜ cffi-mixed git:(main) pip --version pip 21.1.1 from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/pip (python 3.8) (venv) ➜ cffi-mixed git:(main) tox --version 3.24.5 imported from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/tox/init.py


What is interesting is that if I run `maturin develop` before running `tox --skip-missing-interpreters` then the test works.

env) ➜ cffi-mixed git:(main) maturin develop 🍹 Building a mixed python/rust project 🐍 Using CPython 3.8 at /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python to generate the cffi bindings 📡 Using build options bindings from pyproject.toml Collecting cffi Using cached cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB) Collecting pycparser Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB) Installing collected packages: pycparser, cffi Successfully installed cffi-1.15.0 pycparser-2.21 WARNING: You are using pip version 21.1.1; however, version 22.0.3 is available. You should consider upgrading via the '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python -m pip install --upgrade pip' command. Compiling cffi-mixed v0.1.0 (/home/stusmall/Workspace/maturin/test-crates/cffi-mixed) Finished dev [unoptimized + debuginfo] target(s) in 0.55s 📦 Built wheel to /tmp/.tmpsizYQW/cffi_mixed-0.1.0-py3-none-linux_x86_64.whl 🛠 Installed cffi-mixed-0.1.0 (venv) ➜ cffi-mixed git:(main) tox --skip-missing-interpreters py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36 SKIPPED: InterpreterNotFound: python3.6 py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37 SKIPPED: InterpreterNotFound: python3.7 py38 inst-nodeps: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1 py38 run-test-pre: PYTHONHASHSEED='2429669310' py38 run-test: commands[0] | pytest ================================================================================================ test session starts ================================================================================================ platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0 cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache rootdir: /home/stusmall/Workspace/maturin collected 3 items

test_cffi_mixed.py ... [100%]

================================================================================================= 3 passed in 0.02s ================================================================================================= __ summary __ SKIPPED: py36: InterpreterNotFound: python3.6 SKIPPED: py37: InterpreterNotFound: python3.7 py38: commands succeeded congratulations :)



So before I go further, I want to repeat, I'm not a python dev.  So this might be missing the point of the initial bug or misunderstanding how modules work.  But it looks to me that when `from .cffi_mixed import ffi, lib` is run instead of importing from cffi_mixed the wheel it is opening the directory on disk cff_mixed.  When this happens before maturin develop is run, it doesn't have native.so in place, but it does afterwards and so the tests pass
messense commented 2 years ago

@stusmall Sorry for the late reply, I think your hypothesis is correct. We've changed the default project template to include a python folder in #855 to avoid such issue.