Closed ubyndr closed 2 months ago
ebezzi
commented
3 months ago Hey @ubyndr,
we're able to reproduce the issue on our side - we will work with the TileDB team for a fix.
Would you be willing to use conda instead of pypi for this project? If so, we may have a quicker solution available. Let me know.
ubyndr
commented
3 months ago Hi @ebezzi,
Thank you for your prompt response. I appreciate it. Could you please provide more details about the conda solution? I'm particularly interested in understanding how it works and how it can be applied to our current situation.
ebezzi
commented
3 months ago Here is a Dockerfile that should work on Linux/ARM64:
FROM python:3.11
SHELL ["/bin/bash", "-c"]
RUN apt-get update && \
apt-get install -y build-essential zip unzip tar ninja-build && \
apt-get clean && rm -rf /var/lib/apt/lists/*
WORKDIR /app
# required for libtiledb source build
ENV VCPKG_FORCE_SYSTEM_BINARIES=1
ENV VENV="/opt/env"
ENV PATH="$VENV/bin:$PATH"
RUN python3 -m venv $VENV
RUN pip3 install --upgrade pip wheel
RUN pip3 install cellxgene-censusLet me know if you into any other issue.
ubyndr
commented
3 months ago Hi @ebezzi , Thank you for providing the Dockerfile example. I couldn't managed to build the image in my linux environment nor in my github action runner. It fails after almost 2 hours in the runner with the following error message;
ebezzi
commented
3 months ago I believe it could be an issue with python 3.9 - are you able to try with python 3.11 or is there any version constraint that requires you to use 3.9? If so, let me know and I'll do more investigation.
ihnorton
commented
3 months ago I believe it could be an issue with python 3.9
I was not able to build in the 3.9 image due to the older gcc version, so I switched to the 3.11 image.
ubyndr
commented
3 months ago I've also tried it with python 3.11, it still fails;
ebezzi
commented
3 months ago @ubyndr can you share the latest Dockerfile you used for your build, so I can try to reproduce the issue on my own? In the meanwhile, I am looking into an alternative setup with Conda.
ubyndr
commented
3 months ago I've used the Dockerfile you provided;
FROM python:3.11
SHELL ["/bin/bash", "-c"]
RUN apt-get update && \
apt-get install -y build-essential zip unzip tar ninja-build && \
apt-get clean && rm -rf /var/lib/apt/lists/*
WORKDIR /app
# required for libtiledb source build
ENV VCPKG_FORCE_SYSTEM_BINARIES=1
ENV VENV="/opt/env"
ENV PATH="$VENV/bin:$PATH"
RUN python3 -m venv $VENV
RUN pip3 install --upgrade pip wheel
RUN pip3 install cellxgene-censusHi @ubyndr ,
apologies for the delay in the response. I am not quite sure why that Dockerfile doesn't work for you (likely an issue with cross-compilation on your specific machine), but we now have a Conda release of tiledbsoma that comes with prebuilt binaries (and should therefore not cause issues at build time):
FROM condaforge/miniforge3
RUN export MAMBA_NO_BANNER=1
RUN mamba install -c jdblischak/label/experimental -c conda-forge -c tiledb tiledbsoma-py=1.9.5
RUN pip install cellxgene-censusLet me know if this is any help.
ubyndr
commented
2 months ago Thank you @ebezzi , I've managed to build the image. Appreciated.
ebezzi
commented
2 months ago Glad to hear that you managed to build it, @ubyndr ! I'll close this issue, but feel free to reach out if you have any other questions.
Describe the Bug
While attempting to install cellxgene-census within a Docker image built from the python:3.9 base image for the linux/arm64 architecture, the installation process fails. The Dockerfile includes steps to update the system, install build essentials, set up a Python virtual environment, and upgrade pip and wheel before attempting to install cellxgene-census. Despite these preparatory steps, the cellxgene-census installation does not complete successfully. This issue has been observed consistently in the context of building an ARM64-compatible Docker image, suggesting a potential compatibility or dependency resolution problem with the cellxgene-census package or its dependencies on the ARM64 architecture.
To Reproduce
Steps to reproduce the behavior:
Create a Dockerfile with the following content:
Build the Docker container:
Expected Behavior
Being able to successfully build the image.
Environment
Host:
Additional Context
Error message
```txt 299.3 Building wheels for collected packages: tiledbsoma, tiledb, session-info 299.3 Building wheel for tiledbsoma (pyproject.toml): started 371.5 Building wheel for tiledbsoma (pyproject.toml): still running... 371.7 Building wheel for tiledbsoma (pyproject.toml): finished with status 'error' 371.8 error: subprocess-exited-with-error 371.8 371.8 × Building wheel for tiledbsoma (pyproject.toml) did not run successfully. 371.8 │ exit code: 1 371.8 ╰─> [81 lines of output] 371.8 running bdist_wheel 371.8 Build with TileDB: /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/dist 371.8 Building Release build 371.8 -- Using default install prefix /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/dist. To control CMAKE_INSTALL_PREFIX, set OVERRIDE_INSTALL_PREFIX=OFF 371.8 -- Install prefix is /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/dist. 371.8 -- The C compiler identification is GNU 12.2.0 371.8 -- The CXX compiler identification is GNU 12.2.0 371.8 -- Detecting C compiler ABI info 371.8 -- Detecting C compiler ABI info - done 371.8 -- Check for working C compiler: /usr/bin/cc - skipped 371.8 -- Detecting C compile features 371.8 -- Detecting C compile features - done 371.8 -- Detecting CXX compiler ABI info 371.8 -- Detecting CXX compiler ABI info - done 371.8 -- Check for working CXX compiler: /usr/bin/c++ - skipped 371.8 -- Detecting CXX compile features 371.8 -- Detecting CXX compile features - done 371.8 -- Starting TileDB-SOMA superbuild. 371.8 -- Could NOT find TileDB (missing: TileDB_DIR) 371.8 -- Adding TileDB as an external project 371.8 -- Could NOT find spdlog (missing: spdlog_DIR) 371.8 -- Adding spdlog as an external project 371.8 -- Not found clang-tidy 371.8 -- Not found clang-format 371.8 -- Configuring done (8.7s) 371.8 -- Generating done (0.1s) 371.8 -- Build files have been written to: /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build 371.8 [ 4%] Creating directories for 'ep_tiledb' 371.8 [ 8%] Performing download step (download, verify and extract) for 'ep_tiledb' 371.8 -- ep_tiledb download command succeeded. See also /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/src/ep_tiledb-stamp/ep_tiledb-download-*.log 371.8 [ 12%] No update step for 'ep_tiledb' 371.8 [ 16%] No patch step for 'ep_tiledb' 371.8 [ 20%] Performing configure step for 'ep_tiledb' 371.8 CMake Error at /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-Release.cmake:49 (message): 371.8 Command failed: 1 371.8 371.8 '/tmp/pip-build-env-ua2anfxx/overlay/lib/python3.9/site-packages/cmake/data/bin/cmake' '-DCMAKE_INSTALL_PREFIX=/tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/install' '-DCMAKE_PREFIX_PATH=/tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/install' '-DTILEDB_S3=ON' '-DTILEDB_AZURE=ON' '-DTILEDB_GCS=OFF' '-DTILEDB_HDFS=OFF' '-DTILEDB_SERIALIZATION=ON' '-DTILEDB_WERROR=OFF' '-DTILEDB_VERBOSE=OFF' '-DTILEDB_TESTS=OFF' '-DCMAKE_BUILD_TYPE=Release' '-DCMAKE_OSX_ARCHITECTURES=' '-DCMAKE_C_FLAGS=' '-DCMAKE_CXX_FLAGS=' '-DCMAKE_CXX_COMPILER=/usr/bin/c++' '-DCMAKE_C_COMPILER=/usr/bin/cc' '' '-DCMAKE_POSITION_INDEPENDENT_CODE=ON' '-GUnix Makefiles' '-S' '/tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/src/ep_tiledb' '-B' '/tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/src/ep_tiledb-build' 371.8 371.8 See also 371.8 371.8 /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-*.log 371.8 371.8 371.8 gmake[2]: *** [CMakeFiles/ep_tiledb.dir/build.make:92: externals/src/ep_tiledb-stamp/ep_tiledb-configure] Error 1 371.8 gmake[1]: *** [CMakeFiles/Makefile2:89: CMakeFiles/ep_tiledb.dir/all] Error 2 371.8 gmake: *** [Makefile:91: all] Error 2 371.8 Checking: /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/dist/lib/libtiledbsoma.so exists: False 371.8 Checking: /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/dist/lib64/libtiledbsoma.so exists: False 371.8 Checking: /tmp/pip-install-4qk2z4gg/tiledbsoma_31557d620c684f41ade37a7526288fef/dist_links/dist/lib/x86_64-linux-gnu/libtiledbsoma.so exists: False 371.8 libtiledbsoma.so: cannot open shared object file: No such file or directory 371.8 Traceback (most recent call last): 371.8 File "/opt/env/lib/python3.9/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in