The centralized nightlies job failed on Wednesday (2024-08-14)

[github-actions[bot]]

The centralized nightlies job failed on Wednesday (2024-08-14) in run 10397650797

[jdblischak]

The TileDB-MariaDB build failed at the step that confirms the linking with ldd during the tests. There was no error message, so unclear what exactly happened. I restarted it

[github-actions[bot]]

The centralized nightlies job failed on Thursday (2024-08-15) in run 10397650797

[jdblischak]

It failed again during the tests. It just stalls. And unfortunately it erases the log for some reason. The upstream TileDB-MariaDB nightlies are passing.

[jdblischak]

It also failed in my test build against TileDB 2.26.0-rc0 (https://github.com/jdblischak/centralized-tiledb-nightlies/commit/7de75e4cffa49487c28cfc6a39ea8d2afdc80f11, build)

This time I copy-pasted the test output when it stalled:

Logging: /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/mysql-test/mariadb-test-run.pl  --suite=mytile --debug
VS config: 
Path length (125) is longer than maximum supported length (108) and will be truncated at /usr/lib/x86_64-linux-gnu/perl-base/Socket.pm line 193.
Too long tmpdir path '/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/builddir/mysql-test/var/tmp'  creating a shorter one...
 - using tmpdir: '/tmp/_adyXPSPbK'
vardir: /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/builddir/mysql-test/var
Removing old var directory...
Creating var directory '/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/builddir/mysql-test/var'...
Checking supported features...
MariaDB Version 11.0.2-MariaDB-debug
 - SSL connections supported
 - binaries are debug compiled
Using suites: mytile
Collecting tests...
Installing system database...
==============================================================================
TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
mytile.1st                               [ pass ]     16
mytile.2nd                               [ pass ]    239
mytile.aggregates                        [ pass ]    850
mytile.create_allow_subset_existing_array [ pass ]    113
mytile.csv                               [ pass ]     83
mytile.data_types                        [ pass ]   1382
mytile.datetime_pushdown                 [ pass ]   1083
mytile.datetimes                         [ pass ]   1427
mytile.dense_writes                      [ pass ]     72
mytile.discovery                         [ pass ]    232
mytile.duplicates                        [ pass ]    145
mytile.empty_records                     [ pass ]     37
mytile.encryption                        [ pass ]     52
mytile.enum                              [ pass ]    222
mytile.estimated_table_records           [ pass ]    124
mytile.fill_in                           [ pass ]    155
mytile.filters                           [ pass ]    318
mytile.flushing_arrays                   [ pass ]    230
mytile.heterogeneus_dimensions           [ pass ]    140
mytile.hilbert                           [ pass ]   1490
mytile.indexes                           [ pass ]    942
mytile.insert_into_select                [ pass ]    149
mytile.join                              [ pass ]    187
mytile.metadata                          [ pass ]     89
mytile.mrr                               [ pass ]    283
mytile.mrr_datetime_dimensions           [ pass ]    260
mytile.mrr_heterogeneus_dimensions       [ pass ]    234
mytile.mrr_incomplete_queries            [ pass ]    303
mytile.mrr_string_dim                    [ pass ]    470
mytile.mrr_triple_join                   [ pass ]  577338
mytile.multi_value_fixed_length          [ pass ]    109
mytile.nullable_attributes               [ pass ]   1715
mytile.open_at                           [ pass ]    141
mytile.orderby                           [ pass ]     74
mytile.primary_key                       [ pass ]    164
mytile.pushdown_ranges                   [ pass ]    334
mytile.pushdown_two_predicates           [ pass ]    224
mytile.query_conditions                  [ pass ]    330
mytile.schema_evolution                  [ pass ]    196
mytile.spatial_index                     [ pass ]    177
mytile.string_dim                        [ pass ]    231

[jdblischak]

And the tests stalled when running against TileDB 2.25.0 (build)

[jdblischak]

@DimitrisStaratzis do you have any ideas why the nightly TileDB-MariaDB tests are stalling here (against libtiledb dev, 2.26.0-rc0, and 2.25.0) but not in the upstream nightly build?

[github-actions[bot]]

The centralized nightlies job failed on Thursday (2024-08-15) in run 10413538970

[DimitrisStaratzis]

This is very weird, I will check

[DimitrisStaratzis]

Could be related to the fact that MariaDB runs in debug mode in your builds and we have introduced some performance tests like the mrr_triple_join. Could you try building MariaDB in release mode ?

[jdblischak]

Could be related to the fact that MariaDB runs in debug mode in your builds and we have introduced some performance tests like the mrr_triple_join.

@DimitrisStaratzis Good idea! That is a clear difference between this nightly build and the upstream one. Though I am surprised that this problem is only surfacing now, when the upstream nightly build was updated 6 months ago (blame, diff).

Could you try building MariaDB in release mode ?

Testing now in https://github.com/jdblischak/centralized-tiledb-nightlies/commit/b28137e948cd2c7b2e02f8f19f48866d206c5e1e 🤞

[jdblischak]

Confirmed! By applying the changes from https://github.com/TileDB-Inc/TileDB-MariaDB/pull/304 to use Release mode instead of Debug mode, the tests are now passing.

Closed by https://github.com/jdblischak/centralized-tiledb-nightlies/commit/b28137e948cd2c7b2e02f8f19f48866d206c5e1e

[DimitrisStaratzis]

Great! It is weird why it stalls in debug mode though, I will investigate!