search

TileDB-Inc / tiledbsoma-feedstock

A conda-smithy repository for tiledbsoma.
BSD 3-Clause "New" or "Revised" License
3 stars 3 forks source link

tiledbsoma 1.12.0 pre-check #171

Closed johnkerl closed 2 weeks ago

johnkerl commented 3 weeks ago

Following our established procedure

Significant dialog is on https://github.com/single-cell-data/TileDB-SOMA/pull/2692

Context: https://github.com/single-cell-data/TileDB-SOMA/issues/2672

johnkerl commented 3 weeks ago

@jdblischak thanks; I pushed when I'd only done half the work 😓

Pins updated on second commit

johnkerl commented 3 weeks ago

P.S. conda smithy rerender --commit auto says it's up to date ✨

johnkerl commented 2 weeks ago
johnkerl commented 2 weeks ago

Still

Could not solve for environment specs
The following packages are incompatible
├─ aws-crt-cpp 0.26.8.* h1c89d3c_11 is requested and can be installed;
└─ libtiledbsoma 1.12.0.* h9c78a60_0 is not installable because it requires
   └─ tiledb >=2.24.0,<2.25.0a0 , which requires
      └─ aws-crt-cpp >=0.26.10,<0.26.11.0a0 , which conflicts with any installable versions previously reported.

Notes:

👀

jdblischak commented 2 weeks ago

Interesting. So from the logs, we know:

In other words, the new migration to aws-crt-cpp 0.26.12 is not related to this solver error. Looking at the migration status page, no PRs have been opened yet

image

johnkerl commented 2 weeks ago

@jdblischak weerd :|

johnkerl commented 2 weeks ago

Oh wait -- above that in the log:

2024-06-13T14:18:06.1806950Z WARNING: failed to get package records, retrying.  exception was: Unsatisfiable dependencies for platform osx-arm64: {MatchSpec("libtiledbsoma==1.12.0=hdc89182_0"), MatchSpec("tiledb[version='>=2.24.0,<2.25.0a0']")}
2024-06-13T14:18:06.1808150Z Encountered problems while solving:
2024-06-13T14:18:06.1808810Z   - package libtiledbsoma-1.12.0-hdc89182_0 requires tiledb >=2.24.0,<2.25.0a0, but none of the providers can be installed

👀 🤔

jdblischak commented 2 weeks ago

I bet it is only on osx-* because of these pins:

https://github.com/TileDB-Inc/tiledbsoma-feedstock/blob/a44ad933e5fea433a2810f3a716a7868d9984bff/recipe/meta.yaml#L94-L96

The conda-forge migrations are no longer backported to pyarrow/libarrow 12. The oldest version that still receives migrations is libarrow 13

https://github.com/conda-forge/arrow-cpp-feedstock/blob/2f976c55060605a8f6ff265d07e5ce55dad8231b/conda-forge.yml#L4-L8

Looking at the history of PRs I've sent to arrow-cpp-feedstock, I see that I was the one who last migrated 12.x to aws-crt-cpp 0.26.8 (https://github.com/conda-forge/arrow-cpp-feedstock/pull/1377)

image

I can try to manually backport the latest migrations to libarrow 12.x

johnkerl commented 2 weeks ago

Garg. @jdblischak that MacOS/pyarrow bug is killing us in more than one way now. :(

There are several links here:

https://github.com/single-cell-data/TileDB-SOMA/pull/2692#issuecomment-2155015688

cc @ihnorton

jdblischak commented 2 weeks ago

🤞 https://github.com/conda-forge/arrow-cpp-feedstock/pull/1443

h-vetinari commented 2 weeks ago

It's not a viable strategy to keep pyarrow <13 on osx. You should figure out the issues, report it where necessary (upstream or conda-forge), and help get the issue fixed so that the cap can be removed ASAP.

Given the error originates from aws-c-common, I also doubt that it's related to the pyarrow version...🤷 You're in luck that the 12.x branch can still be rerendered and rebuilt, but this will eventually reach a point where the effort to fix becomes prohibitive (and I won't merge any other non-trivial changes - e.g. additional patch backports - other than updating the pins that come from the rerender)

johnkerl commented 2 weeks ago

This solves OK but now I'm trying to isolate a segfault: please see https://github.com/single-cell-data/TileDB-SOMA/pull/2692#issuecomment-2168628050

johnkerl commented 2 weeks ago

It's not a viable strategy to keep pyarrow <13 on osx. You should figure out the issues, report it where necessary (upstream or conda-forge), and help get the issue fixed so that the cap can be removed ASAP.

Given the error originates from aws-c-common, I also doubt that it's related to the pyarrow version...🤷 You're in luck that the 12.x branch can still be rerendered and rebuilt, but this will eventually reach a point where the effort to fix becomes prohibitive (and I won't merge any other non-trivial changes - e.g. additional patch backports - other than updating the pins that come from the rerender)

@h-vetinari thank you for your feedback, and thanks very much for your help here. I am working this as a top/unbreak priority. FWIW, some recent status is at https://github.com/single-cell-data/TileDB-SOMA/pull/2692#issuecomment-2168628050.

Given the error originates from aws-c-common, I also doubt that it's related to the pyarrow version

As noted in my link just above, this is 100% reproducible on MacOS by flipping back and forth between pip install pyarrow==12 (no crash) and pip install pyarrow==13 (crash).

johnkerl commented 2 weeks ago

@h-vetinari please also see https://github.com/apache/arrow/issues/42154

johnkerl commented 2 weeks ago

Status as of commit https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/171/commits/1dce736cc700b908b2414ba5fce0694bcd8a4aa0:

johnkerl commented 2 weeks ago

numpy 2 was released yesterday

johnkerl commented 2 weeks ago

All ☘️ now -- closing this pre-check PR per our established procedure.