The centralized nightlies job failed on Thursday (2024-05-30)

[github-actions[bot]]

The centralized nightlies job failed on Thursday (2024-05-30) in run 9295903705

[jdblischak]

Multiple failures last night: TileDB-MariaDB, TileDB-SOMA-Py, and TileDB-SOMA-R. I re-ran the failed jobs to see if they reproducibly fail

[jdblischak]

I also manually triggered a new build to get the latest commit to TileDB from this morning

[jdblischak]

The TileDB-MariaDB tests failed:

mytile.mrr_triple_join                   [ fail ]
        Test ended at 2024-05-30 02:39:22

CURRENT_TEST: mytile.mrr_triple_join
mysqltest: At line 14: query 'SELECT count(*)
FROM obs JOIN data ON data.soma_dim_0 = obs.soma_joinid
JOIN var ON data.soma_dim_1 = var.soma_joinid
WHERE obs.cell_type = 'T-cells'
AND var.ensembl_gene_name = 'GAPDH'' failed: ER_UNKNOWN_ERROR (1105): [index_read_scan] error for table /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/storage/mytile/mysql-test/mytile/test_data/obs : [TileDB::Task] Error: Caught std::exception: [TileDB::Buffer] Error: Read buffer overflow; may not read beyond buffer size

The result from queries just before the failure was:
CREATE TABLE obs ENGINE=mytile uri='MTR_SUITE_DIR/test_data/obs';;
CREATE TABLE var ENGINE=mytile uri='MTR_SUITE_DIR/test_data/var';;
CREATE TABLE data ENGINE=mytile uri='MTR_SUITE_DIR/test_data/data';;
SET mytile_mrr_support=1;
set optimizer_switch='mrr=on,mrr_sort_keys=on';
set join_cache_level= 6;
set join_buffer_size = 41943040;
set join_buffer_space_limit = 10485[76](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9295903705/job/25583902765#step:8:77)00;
SELECT count(*)
FROM obs JOIN data ON data.soma_dim_0 = obs.soma_joinid
JOIN var ON data.soma_dim_1 = var.soma_joinid
WHERE obs.cell_type = 'T-cells'
AND var.ensembl_gene_name = 'GAPDH';

[jdblischak]

The TileDB-SOMA-Py tests failed with a segmentation fault:

apis/python/tests/test_metadata.py ..................................... [ 45%]
.......................................                                  [ 48%]
Fatal Python error: Segmentation fault

# full traceback omitted

/home/runner/work/_temp/bc11999f-76f5-4cec-b1b4-1f6e9c29762d.sh: line 9:  2179 Segmentation fault      (core dumped) PYTHONPATH=$(pwd)/apis/python/src python -m pytest apis/python/tests
apis/python/tests/test_notebook_sparse_dense.py 

[jdblischak]

The TileDB-SOMA-R tests timed out after the job reached the 6 hour limit:

Rscript -e 'testthat::test_package("tiledbsoma")'
Error: The operation was canceled.

[jdblischak]

Multiple failures last night: TileDB-MariaDB, TileDB-SOMA-Py, and TileDB-SOMA-R. I re-ran the failed jobs to see if they reproducibly fail

Confirmed. They are reproducible test failures https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9295903705/attempts/2

[github-actions[bot]]

The centralized nightlies job failed on Thursday (2024-05-30) in run 9295903705

[jdblischak]

I also manually triggered a new build to get the latest commit to TileDB from this morning

Not only did all the same jobs fail, but now TileDB-Go is also failing. I restarted it, and it failed again.

FAIL    github.com/TileDB-Inc/TileDB-Go [build failed]
FAIL    github.com/TileDB-Inc/TileDB-Go/examples [build failed]

[jdblischak]

PR https://github.com/TileDB-Inc/TileDB/pull/5028 may fix the test failures

[jdblischak]

Manually triggered a build using the PR branch revert-4988-rr/sc-47328-filter-pipeline-output-datatype

https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9305558014/job/25612776659#step:2:70

[jdblischak]

Everything but TileDB-Go is passing now. But the TileDB-Go failure started this morning, not last night, so it is likely unrelated. Closing this now since the main problem has been fixed. If TileDB-Go fails again tonight, it can open a new Issue

[jdblischak]

Also, the TileDB-Go failure may be related to #7