johnkerl
commented
7 months ago As mentioned on Slack (giving a summary here) I cannot reproduce this problem despite my best attempts ...
Closed bkmartinjr closed 6 months ago
johnkerl
commented
7 months ago As mentioned on Slack (giving a summary here) I cannot reproduce this problem despite my best attempts ...
bkmartinjr
commented
7 months ago As mentioned on Slack (giving a summary here) I cannot reproduce this problem despite my best attempts ...
This is a different but than we are discussing in Slack. The above issue is installation in editable mode, e.g., pip install -e ..., which trivially reproduces as a failure.
The slack discussion is about installing in non-editable mode.
johnkerl
commented
7 months ago @bkmartinjr thanks for straightening me out! I had been confused.
ryan-williams
commented
6 months ago Here is a GHA which just successfully ran:
pip install -e 'git+https://github.com/single-cell-data/TileDB-SOMA.git@${{ matrix.ref }}#egg=tiledbsoma&subdirectory=apis/python/'as well as the non-editable version (omitting the -e flag), on main and release-1.[789] branches.
All combinations passed:
I also locally built two Dockerfiles (with and without vcpkg) that exercise the pip install -e git+https:// command:
Presence/Absence of vcpkg was my only idea (based on recent experience on #2244); Github Actions typically has vcpkg installed by default, which can mask problems that show up in local/EC2 builds, but so far I'm unable to reproduce @bkmartinjr's issue.
Possible this was transient / since resolved?
johnkerl
commented
6 months ago @bkmartinjr would you have a few minutes to verify on your setup? 🙏
ryan-williams
commented
6 months ago Sorry, just remembered I did see some failures locally before the successes above, e.g.:
docker run --rm -it --entrypoint pip python:3.11.8 install -e 'git+https://github.com/single-cell-data/TileDB-SOMA#egg=tiledbsoma&subdirectory=apis/python/'That stack looks more like VCPKG-associated issues I saw on #2244 than anything in @bkmartinjr's stack above.
This bit in particular is familiar:
[ 20%] Performing configure step for 'ep_tiledb'
CMake Error at /src/tiledbsoma/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-Release.cmake:49 (message):
Command failed: 1
'/tmp/pip-build-env-z3ciq2tv/overlay/lib/python3.11/site-packages/cmake/data/bin/cmake' '-DCMAKE_INSTALL_PREFIX=/src/tiledbsoma/build/externals/install' '-DCMAKE_PREFIX_PATH=/src/tiledbsoma/build/externals/install' '-DTILEDB_S3=ON' '-DTILEDB_AZURE=ON' '-DTILEDB_GCS=OFF' '-DTILEDB_HDFS=OFF' '-DTILEDB_SERIALIZATION=ON' '-DTILEDB_WERROR=OFF' '-DTILEDB_VERBOSE=OFF' '-DTILEDB_TESTS=OFF' '-DCMAKE_BUILD_TYPE=Release' '-DCMAKE_OSX_ARCHITECTURES=' '-DCMAKE_C_FLAGS=' '-DCMAKE_CXX_FLAGS=' '-DCMAKE_CXX_COMPILER=/usr/bin/c++' '-DCMAKE_C_COMPILER=/usr/bin/cc' '' '-DCMAKE_POSITION_INDEPENDENT_CODE=ON' '-GUnix Makefiles' '-S' '/src/tiledbsoma/build/externals/src/ep_tiledb' '-B' '/src/tiledbsoma/build/externals/src/ep_tiledb-build'
See also
/src/tiledbsoma/build/externals/src/ep_tiledb-stamp/ep_tiledb-configure-*.log
gmake[2]: *** [CMakeFiles/ep_tiledb.dir/build.make:92: externals/src/ep_tiledb-stamp/ep_tiledb-configure] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:89: CMakeFiles/ep_tiledb.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2I will keep investigating.
ryan-williams
commented
6 months ago I'm unable to repro the issue; pip install -e git+https://… seems to work in several contexts I tried.
Here is a sweep of {ubuntu, x86 mac, arm mac} x {main, release-1.[789]}:
That workflow just runs setup-python and pip install -e "git+https…"
runs-on: ${{ matrix.runs-on }}
steps:
- uses: actions/setup-python@v5
with:
python-version: ${{ inputs.python-version }}
- run: |
pip install${{ matrix.editable }} "git+https://github.com/single-cell-data/TileDB-SOMA.git@${{ matrix.ref }}#egg=tiledbsoma&subdirectory=apis/python/"My previous failures were missing system deps required by vcpkg (In a python:3.11.8 Docker image); tiledbsoma's install process can automatically install vcpkg, but assumes some system deps. Here's a working Dockerfile:
FROM python:3.11.8
RUN apt-get update -y && apt install -y cmake curl g++ git make ninja-build tar unzip zip
ARG editable=-e
ARG ref=main
RUN pip install ${editable} "git+https://github.com/single-cell-data/TileDB-SOMA@${ref}#egg=tiledbsoma&subdirectory=apis/python/"Here's a GHA verifying those commands on {main, release-1.[789]} x {'-e',''}:
(python:3.11.8 base image, ubuntu-latest host)
docker build issues with x86 base image on ARM MacsThe only issue I found is: I'm unable to build the Dockerfile above (with its Ubuntu base image) on ARM Macs. export DOCKER_DEFAULT_PLATFORM=linux/amd64 is required to prevent a quick failure during vcpkg install, but then the builds are slow to the point of hanging. Here's one that's been using ≈0% CPU for several hours:
I've seen other x86 Docker images fail to build on ARM Macs, though, so I'm not too worried about that.
I also suspect @bkmartinjr's original issue was resolved at some point since it was filed.
bkmartinjr
commented
6 months ago @ryan-williams - thanks for the detailed exploration. I'm trying to repro it as well with recent branches.
TL;DR - lets close this bug. I'll reopen if it reoccurs and I figure out a reproducible case.
I was attempting to
pip install -eusing the gihtub repo URL for tiledbsoma. This previously worked (as of December-ish).The same thing happens on the
release-1.8branch.It works fine if you omit the
-e(editable) flag