search

chaquo / chaquopy

Chaquopy: the Python SDK for Android
https://chaquo.com/chaquopy/
MIT License
748 stars 127 forks source link

build-wheel.py for duckdb package failed #1181

Open dhimasya opened 2 weeks ago

dhimasya commented 2 weeks ago

already check some issue, but unable to understand how to solve this error when try to prebuild python package for "duckdb" here is my package meta.yaml

package:
  name: duckdb
  version: "1.0.0"

here is the build command that i run python build-wheel.py --python 3.11 --abi arm64-v8a duckdb

/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -idirafter /home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/requirements/chaquopy/include -idirafter /home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/requirements/chaquopy/include/python3.11 -fPIC -DDUCKDB_PYTHON_LIB_NAME=duckdb -DDUCKDB_EXTENSION_PARQUET_LINKED -DDUCKDB_EXTENSION_ICU_LINKED -DDUCKDB_EXTENSION_FTS_LINKED -DDUCKDB_EXTENSION_TPCH_LINKED -DDUCKDB_EXTENSION_TPCDS_LINKED -DDUCKDB_EXTENSION_JSON_LINKED -DDUCKDB_EXTENSION_JEMALLOC_LINKED -DDUCKDB_EXTENSION_AUTOLOAD_DEFAULT=1 -DDUCKDB_EXTENSION_AUTOINSTALL_DEFAULT=1 -Iduckdb_build/src/include -Iduckdb_build/third_party/concurrentqueue -Iduckdb_build/third_party/fast_float -Iduckdb_build/third_party/fastpforlib -Iduckdb_build/third_party/fmt/include -Iduckdb_build/third_party/fsst -Iduckdb_build/third_party/httplib -Iduckdb_build/third_party/hyperloglog -Iduckdb_build/third_party/jaro_winkler -Iduckdb_build/third_party/jaro_winkler/details -Iduckdb_build/third_party/libpg_query -Iduckdb_build/third_party/libpg_query/include -Iduckdb_build/third_party/lz4 -Iduckdb_build/third_party/mbedtls -Iduckdb_build/third_party/mbedtls/include -Iduckdb_build/third_party/mbedtls/library -Iduckdb_build/third_party/miniz -Iduckdb_build/third_party/pcg -Iduckdb_build/third_party/re2 -Iduckdb_build/third_party/skiplist -Iduckdb_build/third_party/tdigest -Iduckdb_build/third_party/utf8proc -Iduckdb_build/third_party/utf8proc/include -Iduckdb_build/third_party/yyjson/include -Iduckdb_build/extension/parquet/include -Iduckdb_build/third_party/parquet -Iduckdb_build/third_party/thrift -Iduckdb_build/third_party/lz4 -Iduckdb_build/third_party/snappy -Iduckdb_build/third_party/zstd/include -Iduckdb_build/third_party/mbedtls -Iduckdb_build/third_party/mbedtls/include -Iduckdb_build/extension/icu/include -Iduckdb_build/extension/icu/third_party/icu/common -Iduckdb_build/extension/icu/third_party/icu/i18n -Iduckdb_build/extension/fts/include -Iduckdb_build/third_party/snowball/libstemmer -Iduckdb_build/third_party/snowball/runtime -Iduckdb_build/third_party/snowball/src_c -Iduckdb_build/extension/tpch/include -Iduckdb_build/extension/tpch/dbgen/include -Iduckdb_build/extension/tpcds/include -Iduckdb_build/extension/tpcds/dsdgen/include -Iduckdb_build/extension/tpcds/dsdgen/include/dsdgen-c -Iduckdb_build/extension/json/include -Iduckdb_build/extension/jemalloc/include -Iduckdb_build/extension/jemalloc/jemalloc/include -Iduckdb_build -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/src/src/include -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/env/lib/python3.11/site-packages/pybind11/include -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/env/lib/python3.11/site-packages/pybind11/include -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/env/include -I/home/dyaadin/chaquopy_builder/python/include/python3.11 -c duckdb_build/extension/jemalloc/jemalloc/src/prof.cpp -o build/temp.linux_aarch64-cpython-311/duckdb_build/extension/jemalloc/jemalloc/src/prof.o -std=c++11 -g0
2 warnings generated.
/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -idirafter /home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/requirements/chaquopy/include -idirafter /home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/requirements/chaquopy/include/python3.11 -fPIC -DDUCKDB_PYTHON_LIB_NAME=duckdb -DDUCKDB_EXTENSION_PARQUET_LINKED -DDUCKDB_EXTENSION_ICU_LINKED -DDUCKDB_EXTENSION_FTS_LINKED -DDUCKDB_EXTENSION_TPCH_LINKED -DDUCKDB_EXTENSION_TPCDS_LINKED -DDUCKDB_EXTENSION_JSON_LINKED -DDUCKDB_EXTENSION_JEMALLOC_LINKED -DDUCKDB_EXTENSION_AUTOLOAD_DEFAULT=1 -DDUCKDB_EXTENSION_AUTOINSTALL_DEFAULT=1 -Iduckdb_build/src/include -Iduckdb_build/third_party/concurrentqueue -Iduckdb_build/third_party/fast_float -Iduckdb_build/third_party/fastpforlib -Iduckdb_build/third_party/fmt/include -Iduckdb_build/third_party/fsst -Iduckdb_build/third_party/httplib -Iduckdb_build/third_party/hyperloglog -Iduckdb_build/third_party/jaro_winkler -Iduckdb_build/third_party/jaro_winkler/details -Iduckdb_build/third_party/libpg_query -Iduckdb_build/third_party/libpg_query/include -Iduckdb_build/third_party/lz4 -Iduckdb_build/third_party/mbedtls -Iduckdb_build/third_party/mbedtls/include -Iduckdb_build/third_party/mbedtls/library -Iduckdb_build/third_party/miniz -Iduckdb_build/third_party/pcg -Iduckdb_build/third_party/re2 -Iduckdb_build/third_party/skiplist -Iduckdb_build/third_party/tdigest -Iduckdb_build/third_party/utf8proc -Iduckdb_build/third_party/utf8proc/include -Iduckdb_build/third_party/yyjson/include -Iduckdb_build/extension/parquet/include -Iduckdb_build/third_party/parquet -Iduckdb_build/third_party/thrift -Iduckdb_build/third_party/lz4 -Iduckdb_build/third_party/snappy -Iduckdb_build/third_party/zstd/include -Iduckdb_build/third_party/mbedtls -Iduckdb_build/third_party/mbedtls/include -Iduckdb_build/extension/icu/include -Iduckdb_build/extension/icu/third_party/icu/common -Iduckdb_build/extension/icu/third_party/icu/i18n -Iduckdb_build/extension/fts/include -Iduckdb_build/third_party/snowball/libstemmer -Iduckdb_build/third_party/snowball/runtime -Iduckdb_build/third_party/snowball/src_c -Iduckdb_build/extension/tpch/include -Iduckdb_build/extension/tpch/dbgen/include -Iduckdb_build/extension/tpcds/include -Iduckdb_build/extension/tpcds/dsdgen/include -Iduckdb_build/extension/tpcds/dsdgen/include/dsdgen-c -Iduckdb_build/extension/json/include -Iduckdb_build/extension/jemalloc/include -Iduckdb_build/extension/jemalloc/jemalloc/include -Iduckdb_build -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/src/src/include -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/env/lib/python3.11/site-packages/pybind11/include -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/env/lib/python3.11/site-packages/pybind11/include -I/home/dyaadin/chaquopy_builder/server/pypi/packages/duckdb/build/1.0.0/cp311-cp311-android_21_arm64_v8a/env/include -I/home/dyaadin/chaquopy_builder/python/include/python3.11 -c duckdb_build/extension/jemalloc/jemalloc/src/prof_data.cpp -o build/temp.linux_aarch64-cpython-311/duckdb_build/extension/jemalloc/jemalloc/src/prof_data.o -std=c++11 -g0
In file included from duckdb_build/extension/jemalloc/jemalloc/src/pac.cpp:2:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h:54:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/prof_structs.h:4:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/ckh.h:4:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd.h:316:
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd_generic.h:93:24: warning: suggest braces around initialization of subobject [-Wmissing-braces]
                        tsd_t initializer = TSD_INITIALIZER;
                                            ^~~~~~~~~~~~~~~
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd.h:153:9: note: expanded from macro 'TSD_INITIALIZER'
                                TSD_DATA_SLOW_INITIALIZER               \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd.h:124:24: note: expanded from macro 'TSD_DATA_SLOW_INITIALIZER'
    /* tcache_slow */           TCACHE_SLOW_ZERO_INITIALIZER,           \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tcache_types.h:26:39: note: expanded from macro 'TCACHE_SLOW_ZERO_INITIALIZER'
#define TCACHE_SLOW_ZERO_INITIALIZER {0}
                                      ^
In file included from duckdb_build/extension/jemalloc/jemalloc/src/pac.cpp:2:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h:54:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/prof_structs.h:4:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/ckh.h:4:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd.h:316:
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd_generic.h:136:22: warning: suggest braces around initialization of subobject [-Wmissing-braces]
        tsd_t initializer = TSD_INITIALIZER;
                            ^~~~~~~~~~~~~~~
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd.h:153:9: note: expanded from macro 'TSD_INITIALIZER'
                                TSD_DATA_SLOW_INITIALIZER               \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tsd.h:124:24: note: expanded from macro 'TSD_DATA_SLOW_INITIALIZER'
    /* tcache_slow */           TCACHE_SLOW_ZERO_INITIALIZER,           \
                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/tcache_types.h:26:39: note: expanded from macro 'TCACHE_SLOW_ZERO_INITIALIZER'
#define TCACHE_SLOW_ZERO_INITIALIZER {0}
                                      ^
duckdb_build/extension/jemalloc/jemalloc/src/jemalloc.cpp:748:3: error: use of undeclared identifier 'pthread_getaffinity_np'; did you mean 'sched_getaffinity'?
                pthread_getaffinity_np(pthread_self(), sizeof(set), &set);
                ^~~~~~~~~~~~~~~~~~~~~~
                sched_getaffinity
/home/dyaadin/chaquopy_builder/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/sched.h:89:5: note: 'sched_getaffinity' declared here
int sched_getaffinity(pid_t __pid, size_t __set_size, cpu_set_t* __set);
    ^
duckdb_build/extension/jemalloc/jemalloc/src/jemalloc.cpp:785:2: error: use of undeclared identifier 'pthread_getaffinity_np'; did you mean 'sched_getaffinity'?
        pthread_getaffinity_np(pthread_self(), sizeof(set), &set);
        ^~~~~~~~~~~~~~~~~~~~~~
        sched_getaffinity
/home/dyaadin/chaquopy_builder/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/sched.h:89:5: note: 'sched_getaffinity' declared here
int sched_getaffinity(pid_t __pid, size_t __set_size, cpu_set_t* __set);
    ^
duckdb_build/extension/jemalloc/jemalloc/src/jemalloc.cpp:986:19: warning: unused variable 'read_source' [-Wunused-variable]
                static unsigned read_source = 0;
                                ^
duckdb_build/extension/jemalloc/jemalloc/src/jemalloc.cpp:3765:11: warning: unused variable 'old_edata' [-Wunused-variable]
        edata_t *old_edata = emap_edata_lookup(tsd_tsdn(tsd),
                 ^
In file included from duckdb_build/extension/jemalloc/jemalloc/src/pai.cpp:2:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h:54:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/prof_structs.h:4:
In file included from duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/ckh.h:4:

look closer that the error happend here

error: use of undeclared identifier 'pthread_getaffinity_np'; did you mean 'sched_getaffinity'?

after looking for a while, i found that i need to set -lpthread on gcc command, how to achieve this and how to suppress some of the warning?

mhsmith commented 1 week ago

This is a compiler error, so a linker flag wouldn't fix it.

The surrounding code is:

#    if defined(JEMALLOC_HAVE_SCHED_SETAFFINITY)
    sched_getaffinity(0, sizeof(set), &set);
#    else /* !JEMALLOC_HAVE_SCHED_SETAFFINITY */
    pthread_getaffinity_np(pthread_self(), sizeof(set), &set);
#    endif /* JEMALLOC_HAVE_SCHED_SETAFFINITY */

From searching the Android headers ($ANDROID_HOME/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include), I see that Android does have sched_setaffinity, but it doesn't have pthread_getaffinity_np. So the fix is to define JEMALLOC_HAVE_SCHED_SETAFFINITY.

This can be done by editing duckdb_build/extension/jemalloc/jemalloc/include/jemalloc/internal/jemalloc_internal_defs.h to replace /* #undef JEMALLOC_HAVE_SCHED_SETAFFINITY */ with #define JEMALLOC_HAVE_SCHED_SETAFFINITY.

For how to apply and save this edit, see the build-wheel README under "If any changes are needed to make the build work".

mhsmith commented 1 week ago

The build then fails with these errors:

In file included from duckdb_build/ub_src_main_extension.cpp:5:
In file included from duckdb_build/src/main/extension/extension_install.cpp:14:
duckdb_build/third_party/httplib/httplib.hpp:2687:2: error: use of undeclared identifier 'getifaddrs'; did you mean 'ifaddrs'?
        getifaddrs(&ifap);
        ^
/home/smith/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/../sysroot/usr/include/ifaddrs.h:45:8: note: 'ifaddrs' declared here
struct ifaddrs {
       ^
In file included from duckdb_build/ub_src_main_extension.cpp:5:
In file included from duckdb_build/src/main/extension/extension_install.cpp:14:
duckdb_build/third_party/httplib/httplib.hpp:2694:6: error: use of undeclared identifier 'freeifaddrs'
                                        freeifaddrs(ifap);
                                        ^
duckdb_build/third_party/httplib/httplib.hpp:2700:2: error: use of undeclared identifier 'freeifaddrs'
        freeifaddrs(ifap);
        ^
3 errors generated.

Searching the sysroot/usr/include directory again, I found that these functions were added to Android in API level 24, so you can work around this by passing --api-level 24 to build-wheel. To use the resulting wheels, you'll also have to make sure the minSdk of your app is 24 or higher.

There may be other issues; I haven't tested any further.