Build failing in linux/arm64 Ubuntu VM

Hi!

I am experiencing trouble when building TileDB-VCF under the following conditions:

Host: Apple Silicon M2 VM Manager: UTM VM Image OS: Ubuntu 22.04.3 LTS aarch64

Do you have any ideas of what may be the culprit here? Apart from the fact that ld fails, that is. For the record, TileDB appears to install fine via pip install tiledb, just in case that is at all relevant.

NB! Using conda is a no go.

[ 95%] Linking CXX shared library libtiledbvcf.so
[ 97%] Building CXX object src/CMakeFiles/tiledbvcf-bin.dir/cli/tiledbvcf.cc.o
/usr/bin/ld: /TileDB-VCF/libtiledbvcf/build/externals/install/lib/libtiledb.so.2.17: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
make[5]: *** [src/CMakeFiles/tiledbvcf.dir/build.make:157: src/libtiledbvcf.so] Error 1
make[4]: *** [CMakeFiles/Makefile2:147: src/CMakeFiles/tiledbvcf.dir/all] Error 2
make[4]: *** Waiting for unfinished jobs....
[100%] Linking CXX executable tiledbvcf
/usr/bin/ld: /TileDB-VCF/libtiledbvcf/build/externals/install/lib/libtiledb.so.2.17: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
make[5]: *** [src/CMakeFiles/tiledbvcf-bin.dir/build.make:173: src/tiledbvcf] Error 1
make[4]: *** [CMakeFiles/Makefile2:173: src/CMakeFiles/tiledbvcf-bin.dir/all] Error 2
make[3]: *** [Makefile:146: all] Error 2
make[2]: *** [CMakeFiles/libtiledbvcf.dir/build.make:86: libtiledbvcf-prefix/src/libtiledbvcf-stamp/libtiledbvcf-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:228: CMakeFiles/libtiledbvcf.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
Traceback (most recent call last):
  File "/TileDB-VCF/apis/python/setup.py", line 274, in <module>
CMake configure command: ['cmake', '-DFORCE_EXTERNAL_HTSLIB=ON', '-DFORCE_EXTERNAL_TILEDB=OFF', '-DTILEDB_S3=ON', '-DDOWNLOAD_TILEDB_PREBUILT=ON', '-DCMAKE_BUILD_TYPE=Release', '-DENABLE_ARROW_EXPORT=ON', '../../libtiledbvcf']
    setup(
  File "/usr/local/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/usr/local/lib/python3.11/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "/usr/local/lib/python3.11/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "/TileDB-VCF/apis/python/setup.py", line 264, in run
    find_or_build_libtiledbvcf(self)
  File "/TileDB-VCF/apis/python/setup.py", line 179, in find_or_build_libtiledbvcf
    build_libtiledbvcf()
  File "/TileDB-VCF/apis/python/setup.py", line 164, in build_libtiledbvcf
    subprocess.check_call(build_cmd, cwd=build_dir)
  File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j6']' returned non-zero exit status 2.
------
failed to solve: process "/bin/sh -c python setup.py install" did not complete successfully: exit code: 1

Thanks for the issue @dotinspace. It looks like the problem is we always download the x86_64 version of libtiledb.so when the OS is Linux.

Please try this command, which will build TileDB on your system instead of downloading a prebuilt version.

cd apis/python
python setup.py install --disable-download-tiledb-prebuilt

Many thanks! That appears to have solved the issue.

Hmmm... That did work momentarily. Subsequent builds fails, both when trying to build arm64 and amd64. Below are errors:

TileDB failing on arm64 via pip install tiledb

19.83 Building wheels for collected packages: tiledb
19.83   Building wheel for tiledb (pyproject.toml): started
73.43   Building wheel for tiledb (pyproject.toml): finished with status 'error'
73.44   error: subprocess-exited-with-error
73.44   
73.44   × Building wheel for tiledb (pyproject.toml) did not run successfully.
73.44   │ exit code: 1
73.44   ╰─> [96 lines of output]
73.44       <string>:9: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
73.44       WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
73.44       TILEDB_PATH from env: ''
73.44       libtiledb_exists checking 'library_dirs': []
73.44       libtiledb_exists found: 'None'
73.44       running bdist_wheel
73.44       Cloning into '/tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0'...
73.44       Note: switching to '71ca8b4ddc615e2ad124fb76427df65667850e5e'.
73.44       
73.44       You are in 'detached HEAD' state. You can look around, make experimental
73.44       changes and commit them, and you can discard any commits you make in this
73.44       state without impacting any branches by switching back to a branch.
73.44       
73.44       If you want to create a new branch to retain commits you create, you may
73.44       do so (now or later) by using -c with the switch command. Example:
73.44       
73.44         git switch -c <new-branch-name>
73.44       
73.44       Or undo this operation with:
73.44       
73.44         git switch -
73.44       
73.44       Turn off this advice by setting config variable advice.detachedHead to false
73.44       
73.44       GNU Make 4.3
73.44       Built for aarch64-unknown-linux-gnu
73.44       Copyright (C) 1988-2020 Free Software Foundation, Inc.
73.44       License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
73.44       This is free software: you are free to change and redistribute it.
73.44       There is NO WARRANTY, to the extent permitted by law.
73.44       -- Update submodules
73.44       '/usr/bin/git' 'submodule' 'update' '--init' '--recursive'
73.44       Submodule 'external/vcpkg' (https://github.com/microsoft/vcpkg) registered for path 'external/vcpkg'
73.44       Cloning into '/tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0/external/vcpkg'...
73.44       Submodule path 'external/vcpkg': checked out '1b4d69f3028d74401a001aa316986a670ca6289a'
73.44       Using vcpkg features: serialization;s3;webp
73.44       -- Bootstrapping vcpkg before install
73.44       -- Bootstrapping vcpkg before install - failed
73.44       CMake Error at external/vcpkg/scripts/buildsystems/vcpkg.cmake:899 (message):
73.44         vcpkg install failed.  See logs for more information:
73.44         /tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0/build/vcpkg-bootstrap.log
73.44       Call Stack (most recent call first):
73.44         /tmp/pip-build-env-6owl3re7/normal/lib/python3.11/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeDetermineSystem.cmake:148 (include)
73.44         CMakeLists.txt:140 (project)
73.44       
73.44       
73.44       CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
73.44       CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
73.44       CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
73.44       -- Configuring incomplete, errors occurred!
73.44       ext_modules:  [<setuptools.extension.Extension('tiledb.libtiledb') at 0xffff9fc1bc90>, <setuptools.extension.Extension('tiledb.main') at 0xffff9f2980d0>, <pybind11.setup_helpers.Pybind11Extension('tiledb.cc') at 0xffff9f9a45d0>]
73.44       tiledb_ext:  <setuptools.extension.Extension('tiledb.libtiledb') at 0xffff9fc1bc90>
73.44       main_ext:  <setuptools.extension.Extension('tiledb.main') at 0xffff9f2980d0>
73.44       tiledb_cc_ext:  <pybind11.setup_helpers.Pybind11Extension('tiledb.cc') at 0xffff9f9a45d0>
73.44       tiledb_ext.library_dirs:  []
73.44       libtiledb_exists checking 'library_dirs': []
73.44       libtiledb_exists found: 'None'
73.44       Building libtiledb in directory /tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0/build...
73.44       CMake configure command: ['cmake', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0/dist', '-DTILEDB_TESTS=OFF', '-DTILEDB_S3=ON', '-DTILEDB_WERROR=OFF', '-DTILEDB_HDFS=ON', '-DTILEDB_INSTALL_LIBDIR=lib', '-DTILEDB_CPP_API=ON', '-DTILEDB_LOG_OUTPUT_ON_FAILURE=ON', '-DTILEDB_FORCE_ALL_DEPS:BOOL=OFF', '-DTILEDB_SERIALIZATION:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release', '/tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0']
73.44       Traceback (most recent call last):
73.44         File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
73.44           main()
73.44         File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
73.44           json_out['return_val'] = hook(**hook_input['kwargs'])
73.44                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
73.44         File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
73.44           return _build_backend().build_wheel(wheel_directory, config_settings,
73.44                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in build_wheel
73.44           return self._build_with_temp_dir(
73.44                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
73.44           self.run_setup()
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
73.44           exec(code, locals())
73.44         File "<string>", line 786, in <module>
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
73.44           return distutils.core.setup(**attrs)
73.44                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
73.44           return run_commands(dist)
73.44                  ^^^^^^^^^^^^^^^^^^
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
73.44           dist.run_commands()
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
73.44           self.run_command(cmd)
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
73.44           super().run_command(command)
73.44         File "/tmp/pip-build-env-6owl3re7/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
73.44           cmd_obj.run()
73.44         File "<string>", line 506, in run
73.44         File "<string>", line 314, in find_or_install_libtiledb
73.44         File "<string>", line 275, in build_libtiledb
73.44         File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call
73.44           raise CalledProcessError(retcode, cmd)
73.44       subprocess.CalledProcessError: Command '['cmake', '-DCMAKE_INSTALL_PREFIX=/tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0/dist', '-DTILEDB_TESTS=OFF', '-DTILEDB_S3=ON', '-DTILEDB_WERROR=OFF', '-DTILEDB_HDFS=ON', '-DTILEDB_INSTALL_LIBDIR=lib', '-DTILEDB_CPP_API=ON', '-DTILEDB_LOG_OUTPUT_ON_FAILURE=ON', '-DTILEDB_FORCE_ALL_DEPS:BOOL=OFF', '-DTILEDB_SERIALIZATION:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release', '/tmp/pip-install-7t223fh2/tiledb_f2af4fc6de4c4b33a65474d76b321dfd/build/TileDB-2.18.0']' returned non-zero exit status 1.
73.44       [end of output]
73.44   
73.44   note: This error originates from a subprocess, and is likely not a problem with pip.
73.44   ERROR: Failed building wheel for tiledb
73.44 Failed to build tiledb
73.44 ERROR: Could not build wheels for tiledb, which is required to install pyproject.toml-based projects

TileDB-VCF Python api build fails on amd64 via setup.py install --disable-download-tiledb-prebuilt

554.7 CMake Error at /tiledb/ext_lib/TileDB-VCF/libtiledbvcf/build/externals/src/ep_tiledb-stamp/ep_tiledb-install-Release.cmake:49 (message):
554.7   Command failed: 2
554.7 
554.7    '/usr/bin/cmake' '--build' '.' '--target' 'install-tiledb'
554.7 
554.7   See also
554.7 
554.7     /tiledb/ext_lib/TileDB-VCF/libtiledbvcf/build/externals/src/ep_tiledb-stamp/ep_tiledb-install-*.log
554.7 
554.7 
554.7 make[2]: *** [CMakeFiles/ep_tiledb.dir/build.make:104: externals/src/ep_tiledb-stamp/ep_tiledb-install] Error 1
554.7 make[1]: *** [CMakeFiles/Makefile2:173: CMakeFiles/ep_tiledb.dir/all] Error 2
554.7 make: *** [Makefile:91: all] Error 2
554.7 CMake configure command: ['cmake', '-DFORCE_EXTERNAL_HTSLIB=ON', '-DFORCE_EXTERNAL_TILEDB=OFF', '-DTILEDB_S3=ON', '-DDOWNLOAD_TILEDB_PREBUILT=OFF', '-DCMAKE_BUILD_TYPE=Release', '-DENABLE_ARROW_EXPORT=ON', '../../libtiledbvcf']
554.7 Traceback (most recent call last):
554.7   File "/tiledb/ext_lib/TileDB-VCF/apis/python/setup.py", line 274, in <module>
554.7     setup(
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
554.7     return distutils.core.setup(**attrs)
554.7            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
554.7     return run_commands(dist)
554.7            ^^^^^^^^^^^^^^^^^^
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
554.7     dist.run_commands()
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
554.7     self.run_command(cmd)
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
554.7     super().run_command(command)
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
554.7     cmd_obj.run()
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/command/install.py", line 74, in run
554.7     self.do_egg_install()
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/command/install.py", line 123, in do_egg_install
554.7     self.run_command('bdist_egg')
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
554.7     self.distribution.run_command(command)
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
554.7     super().run_command(command)
554.7   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
554.7     cmd_obj.run()
554.7   File "/tiledb/ext_lib/TileDB-VCF/apis/python/setup.py", line 264, in run
554.7     find_or_build_libtiledbvcf(self)
554.7   File "/tiledb/ext_lib/TileDB-VCF/apis/python/setup.py", line 179, in find_or_build_libtiledbvcf
554.7     build_libtiledbvcf()
554.7   File "/tiledb/ext_lib/TileDB-VCF/apis/python/setup.py", line 164, in build_libtiledbvcf
554.7     subprocess.check_call(build_cmd, cwd=build_dir)
554.7   File "/usr/local/lib/python3.11/subprocess.py", line 413, in check_call
554.7     raise CalledProcessError(retcode, cmd)
554.7 subprocess.CalledProcessError: Command '['make', '-j6']' returned non-zero exit status 2.
------
failed to solve: process "/bin/sh -c python setup.py install --disable-download-tiledb-prebuilt" did not complete successfully: exit code: 1

Not sure why it worked nicely a couple of days ago. Any pointers would be greatly appreciated.

Those look like build errors in the main branch we are currently resolving.

Can you try the released versions of TileDB-VCF (0.26.6) and TileDB-Py (0.23.4)?

Unfortunately, that does not work.

0.484 /usr/local/lib/python3.11/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
0.484   warnings.warn(
1.420 Traceback (most recent call last):
1.420   File "/tiledb/ext_lib/TileDB-VCF/apis/python/setup.py", line 274, in <module>
1.420     setup(
1.420   File "/usr/local/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
1.420     return distutils.core.setup(**attrs)
1.420            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.420   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 147, in setup
1.421     _setup_distribution = dist = klass(attrs)
1.421                                  ^^^^^^^^^^^^
1.421   File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 476, in __init__
1.421     _Distribution.__init__(
1.421   File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 282, in __init__
1.421     self.finalize_options()
1.421   File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 899, in finalize_options
1.421     for ep in sorted(loaded, key=by_order):
1.421               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.421   File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 898, in <lambda>
1.421     loaded = map(lambda e: e.load(), filtered)
1.421                            ^^^^^^^^
1.421   File "/usr/local/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
1.421     module = import_module(match.group('module'))
1.421              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.421   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
1.421     return _bootstrap._gcd_import(name[level:], package, level)
1.421            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.421   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
1.421   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
1.421   File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
1.421   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
1.421   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
1.421   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
1.421   File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
1.421   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
1.421   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
1.421   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
1.421   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
1.421   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
1.421   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
1.421   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
1.421   File "/tiledb/ext_lib/TileDB-VCF/apis/python/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/__init__.py", line 7, in <module>
1.421     from ._config import Configuration
1.421   File "/tiledb/ext_lib/TileDB-VCF/apis/python/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/_config.py", line 13, in <module>
1.421     from . import _log
1.421   File "/tiledb/ext_lib/TileDB-VCF/apis/python/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/_log.py", line 45, in <module>
1.421     _default_handler = make_default_handler()
1.421                        ^^^^^^^^^^^^^^^^^^^^^^
1.421   File "/tiledb/ext_lib/TileDB-VCF/apis/python/.eggs/setuptools_scm-8.0.4-py3.11.egg/setuptools_scm/_log.py", line 33, in make_default_handler
1.421     from rich.console import Console
1.421   File "/usr/local/lib/python3.11/site-packages/rich/console.py", line 60, in <module>
1.421     from .pretty import Pretty, is_expandable
1.421   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
1.421   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
1.421   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
1.421   File "<frozen importlib._bootstrap_external>", line 936, in exec_module
1.421   File "<frozen importlib._bootstrap_external>", line 1069, in get_code
1.421   File "<frozen importlib._bootstrap_external>", line 729, in _compile_bytecode
1.421 ValueError: bad marshal data (unknown type code)
------
failed to solve: process "/bin/sh -c python setup.py install --disable-download-tiledb-prebuilt" did not complete successfully: exit code: 1

The ValueError: bad marshal data (unknown type code) error can happen when .pyc files were corrupted or compiled with a different version of python.

You can try reinstalling the rich package, clearing the python cache, or starting from a new virtual environment (if you are using one).

Whoops, I pasted wrong error. Apologies. The problem is when building pyarrow. Do I need a particular version it too?

Example error:

/usr/local/lib/python3.11/site-packages/pyarrow/include/arrow/util/aligned_storage.h:36:37: error: ‘reinterpret_cast’ from type ‘const std::aligned_storage<16, 8>::type*’ to type ‘std::shared_ptr<arrow::Tensor>*’ casts away qualifiers
7976.4 error: command '/usr/bin/gcc' failed with exit code 1

We use pyarrow 9 in our nightly CI tests, but that doesn't build pyarrow from source.

This happens when using the 0.26.6 release as suggested. And it happens during build of the python api for tiledb-vcf. Weird.

TileDB-Inc / TileDB-VCF

Build failing in linux/arm64 Ubuntu VM #623