The centralized nightlies job failed on Tuesday (2024-06-11)

[github-actions[bot]]

The centralized nightlies job failed on Tuesday (2024-06-11) in run 9475617750

[github-actions[bot]]

The centralized nightlies job failed on Wednesday (2024-06-12) in run 9475617750

[jdblischak]

TileDB-Py failed to create a wheel. I restarted the job and it failed reproducibly.

I think this is another numpy Issue. During the standard build (which passed), it installed numpy 1.26.4. But during the wheel build (which failed), it installed numpy 2.0.0

Compiling tiledb/libtiledb.pyx because it depends on /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/.eggs/numpy-2.0.0rc2-py3.11-linux-x86_64.egg/numpy/__init__.cython-30.pxd.

I suspect this is related to https://github.com/TileDB-Inc/TileDB-Py/pull/1989. @kounelisagis do you have any ideas for troubleshooting this?

[github-actions[bot]]

The centralized nightlies job failed on Wednesday (2024-06-12) in run 9492584193

[github-actions[bot]]

The centralized nightlies job failed on Thursday (2024-06-13) in run 9509565633

[jdblischak]

The TileDB-Py wheel build is continuing to fail. There are a bunch of error messages about "not declared in this scope", eg

.eggs/numpy-2.0.0rc2-py3.11-linux-x86_64.egg/numpy/__init__.cython-30.pxd:351:13: error: ‘PyArray_MultiIter_ITERS’ was not declared in this scope; did you mean ‘PyArray_MultiIter_NEXT’?

[github-actions[bot]]

The centralized nightlies job failed on Friday (2024-06-14) in run 9524695087

[github-actions[bot]]

The centralized nightlies job failed on Saturday (2024-06-15) in run 9532591390

[github-actions[bot]]

The centralized nightlies job failed on Sunday (2024-06-16) in run 9540908808

[jdblischak]

• Saturday night's failure was still the TileDB-Py numpy 2 issue
• Sunday night's failure was different. The TileDB-Py build passed! I assume this is because pip installed numpy 2.0.0, especially since there were no relevant commits upstream
• But building TileDB-Py with numpy 2 caused both tiledbvcf-py and tiledbsoma-py to fail their tests due to numpy problems

[github-actions[bot]]

The centralized nightlies job failed on Monday (2024-06-17) in run 9557834549

[jdblischak]

In last night's run, the tiledbsoma-py tests passed. I assume this is due to the recently merged https://github.com/single-cell-data/TileDB-SOMA/commit/6025c8db824a029bf5249e6ef2d318e2ff7e8814 (https://github.com/single-cell-data/TileDB-SOMA/pull/2745), which required numpy<2.0 in the setup.py. Looking at the build log, numpy 2 was initially installed, but then it was downgraded to numpy 1.26.4.

In contrast, tiledbvcf-py continues to fail the import test:

python -c "import tiledbvcf; print(tiledbvcf.version)"
## A module that was compiled using NumPy 1.x cannot be run in
## NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
## versions of NumPy, modules must be compiled with NumPy 2.0.
## Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
## 
## If you are a user of the module, the easiest solution will be to
## downgrade to 'numpy<2' or try to upgrade the affected module.
## We expect that some modules will need time to support NumPy 2.
## 
## Traceback (most recent call last):  File "<string>", line 1, in <module>
##   File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/tiledbvcf/__init__.py", line 2, in <module>
##     import pyarrow
##   File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pyarrow/__init__.py", line 65, in <module>
##     import pyarrow.lib as _lib
## AttributeError: _ARRAY_API not found
## Traceback (most recent call last):
##   File "<string>", line 1, in <module>
##   File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/tiledbvcf/__init__.py", line 2, in <module>
##     import pyarrow
##   File "/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/site-packages/pyarrow/__init__.py", line 65, in <module>
##     import pyarrow.lib as _lib
##   File "pyarrow/lib.pyx", line 37, in init pyarrow.lib
## ImportError: numpy.core.multiarray failed to import

[github-actions[bot]]

The centralized nightlies job failed on Tuesday (2024-06-18) in run 9575041254

[jdblischak]

The tiledbvcf-py build is failing with the same numpy-related import error as before. However, now the tiledbsoma-py tests are failing with the error message tiledb.cc.TileDBError: Group: Cannot open group; Group does not exist.

[jdblischak]

The tiledbsoma-py test failures were fixed by https://github.com/single-cell-data/TileDB-SOMA/pull/2753#pullrequestreview-2128833193 (confirmed in https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9585883108/job/26433519955#step:12:348)

[github-actions[bot]]

The centralized nightlies job failed on Wednesday (2024-06-19) in run 9590553151

[jdblischak]

Still only the tiledbvcf-py import error

[github-actions[bot]]

The centralized nightlies job failed on Thursday (2024-06-20) in run 9607203617

[jdblischak]

The tiledbvcf-py import error should be fixed when the pin numpy<2 is merged upstream in https://github.com/TileDB-Inc/TileDB-VCF/pull/731

[github-actions[bot]]

The centralized nightlies job failed on Friday (2024-06-21) in run 9622093821

[github-actions[bot]]

The centralized nightlies job failed on Saturday (2024-06-22) in run 9630033186

[github-actions[bot]]

The centralized nightlies job failed on Sunday (2024-06-23) in run 9638470261

[jdblischak]

Nightly tiledbvcf-py build should be fixed by https://github.com/TileDB-Inc/TileDB-VCF/commit/f5707027301a2e0910cd2f276419a3101cf4267a (https://github.com/TileDB-Inc/TileDB-VCF/pull/734), confirmed in https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9650757378