search

TileDB-Inc / TileDB-VCF

Efficient variant-call data storage and retrieval library using the TileDB storage library.
https://tiledb-inc.github.io/TileDB-VCF/
MIT License
85 stars 14 forks source link

Add nightly build (linux only for now) #577

Closed jdblischak closed 11 months ago

jdblischak commented 11 months ago

Nightly build against nightly libtiledb. Will add macOS and Windows builds in the future. The main blocker is finding a compatible pyarrow. I've tried multiple versions of pyarrow, installed via either pip or conda, and I can't find an option that successfully builds for both Linux and macOS.

cc: @ihnorton, @awenocur

jdblischak commented 11 months ago

@ihnorton I've added the tests for libtiledbvcf and tiledbvcf-py. One question: should I also build dev TileDB-Py from source? Since I install TileDB-Py from conda, ldd shows it linked to the conda-installed libtiledb (ie not the nightly one I built from source and installed in ./install/)

jdblischak commented 11 months ago

One question: should I also build dev TileDB-Py from source? Since I install TileDB-Py from conda, ldd shows it linked to the conda-installed libtiledb (ie not the nightly one I built from source and installed in ./install/)

I'm less concerned about this now after doing more local testing. While both tiledb.libtiledb.version() and ldd say that the conda-installed TileDB-Py is linked to the conda-installed 2.17.0, which has S3 support built-in during the build, I still have reason to believe that TileDB-Py is accessing the dev .so (2.18.0), presumably because I manually set LD_LIBRARY_PATH. When I don't build dev libtiledb with S3 support, the pytest tests fail with [TileDB::VFS] Error: TileDB was built without S3 support. And when I re-built libtiledb from source with -D TILEDB_S3=ON, then the pytest tests passed.

ihnorton commented 11 months ago

Thanks 💯

When I don't build dev libtiledb with S3 support, the pytest tests fail with

Nice differential!!

One question: should I also build dev TileDB-Py from source? I'm less concerned about this now after doing more local testing.

We could end up with symbol issues at some point, but as long as it works, lets get this going and address this and other issues in follow-up.

jdblischak commented 11 months ago

@ihnorton The nightlies started failing 3 days ago with the dev version of libtiledb, but we forgot to create the new label "nightly-failure", so the failed jobs haven't opened an Issue. Could you please create the label?