Closed github-actions[bot] closed 3 months ago
Multiple failures last night: TileDB-MariaDB, TileDB-SOMA-Py, and TileDB-SOMA-R. I re-ran the failed jobs to see if they reproducibly fail
I also manually triggered a new build to get the latest commit to TileDB from this morning
The TileDB-MariaDB tests failed:
mytile.mrr_triple_join [ fail ]
Test ended at 2024-05-30 02:39:22
CURRENT_TEST: mytile.mrr_triple_join
mysqltest: At line 14: query 'SELECT count(*)
FROM obs JOIN data ON data.soma_dim_0 = obs.soma_joinid
JOIN var ON data.soma_dim_1 = var.soma_joinid
WHERE obs.cell_type = 'T-cells'
AND var.ensembl_gene_name = 'GAPDH'' failed: ER_UNKNOWN_ERROR (1105): [index_read_scan] error for table /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/storage/mytile/mysql-test/mytile/test_data/obs : [TileDB::Task] Error: Caught std::exception: [TileDB::Buffer] Error: Read buffer overflow; may not read beyond buffer size
The result from queries just before the failure was:
CREATE TABLE obs ENGINE=mytile uri='MTR_SUITE_DIR/test_data/obs';;
CREATE TABLE var ENGINE=mytile uri='MTR_SUITE_DIR/test_data/var';;
CREATE TABLE data ENGINE=mytile uri='MTR_SUITE_DIR/test_data/data';;
SET mytile_mrr_support=1;
set optimizer_switch='mrr=on,mrr_sort_keys=on';
set join_cache_level= 6;
set join_buffer_size = 41943040;
set join_buffer_space_limit = 10485[76](https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9295903705/job/25583902765#step:8:77)00;
SELECT count(*)
FROM obs JOIN data ON data.soma_dim_0 = obs.soma_joinid
JOIN var ON data.soma_dim_1 = var.soma_joinid
WHERE obs.cell_type = 'T-cells'
AND var.ensembl_gene_name = 'GAPDH';
The TileDB-SOMA-Py tests failed with a segmentation fault:
apis/python/tests/test_metadata.py ..................................... [ 45%]
....................................... [ 48%]
Fatal Python error: Segmentation fault
# full traceback omitted
/home/runner/work/_temp/bc11999f-76f5-4cec-b1b4-1f6e9c29762d.sh: line 9: 2179 Segmentation fault (core dumped) PYTHONPATH=$(pwd)/apis/python/src python -m pytest apis/python/tests
apis/python/tests/test_notebook_sparse_dense.py
The TileDB-SOMA-R tests timed out after the job reached the 6 hour limit:
Rscript -e 'testthat::test_package("tiledbsoma")'
Error: The operation was canceled.
Multiple failures last night: TileDB-MariaDB, TileDB-SOMA-Py, and TileDB-SOMA-R. I re-ran the failed jobs to see if they reproducibly fail
Confirmed. They are reproducible test failures https://github.com/jdblischak/centralized-tiledb-nightlies/actions/runs/9295903705/attempts/2
The centralized nightlies job failed on Thursday (2024-05-30) in run 9295903705
I also manually triggered a new build to get the latest commit to TileDB from this morning
Not only did all the same jobs fail, but now TileDB-Go is also failing. I restarted it, and it failed again.
FAIL github.com/TileDB-Inc/TileDB-Go [build failed]
FAIL github.com/TileDB-Inc/TileDB-Go/examples [build failed]
PR https://github.com/TileDB-Inc/TileDB/pull/5028 may fix the test failures
Manually triggered a build using the PR branch revert-4988-rr/sc-47328-filter-pipeline-output-datatype
Everything but TileDB-Go is passing now. But the TileDB-Go failure started this morning, not last night, so it is likely unrelated. Closing this now since the main problem has been fixed. If TileDB-Go fails again tonight, it can open a new Issue
Also, the TileDB-Go failure may be related to #7
The centralized nightlies job failed on Thursday (2024-05-30) in run 9295903705