search

jdblischak / centralized-tiledb-nightlies

demo
1 stars 0 forks source link

The centralized nightlies job failed on Tuesday (2024-07-09) #14

Closed github-actions[bot] closed 1 week ago

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Tuesday (2024-07-02) in run 9770774146

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Tuesday (2024-07-09) in run 9867069920

jdblischak commented 1 month ago

Oops. I forgot to close this Issue yesterday. The original failure from last week was a spurious error when uploading an artifact. It has passed ever since then.

jdblischak commented 1 month ago

Last night's failure was due to the recent update to the dump functions

xref: https://github.com/TileDB-Inc/TileDB/pull/5026, https://github.com/TileDB-Inc/conda-forge-nightly-controller/issues/108, https://github.com/TileDB-Inc/TileDB-R/pull/727, https://github.com/TileDB-Inc/TileDB-Py/pull/1975

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Wednesday (2024-07-10) in run 9884378480

jdblischak commented 1 month ago

Still failing due to the dump error

jdblischak commented 1 month ago

Fix pending in https://github.com/TileDB-Inc/TileDB/pull/5179

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Thursday (2024-07-11) in run 9901331468

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Friday (2024-07-12) in run 9916513178

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Saturday (2024-07-13) in run 9924615480

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Sunday (2024-07-14) in run 9932722986

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Monday (2024-07-15) in run 9949626990

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Tuesday (2024-07-16) in run 9967040160

jdblischak commented 1 month ago

Last night libtiledb failed due to a spurious download failure. Restarted

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Wednesday (2024-07-17) in run 9967040160

jdblischak commented 1 month ago

While the libtiledb failure last night was spurious, most of the client builds are now failing.

image

I suspect it is related to the recent removal of deprecated APIs in https://github.com/TileDB-Inc/TileDB/pull/5146

I'm seeing similar error messages to those in other nightly builds

jdblischak commented 1 month ago

xref: https://github.com/TileDB-Inc/TileDB-VCF/issues/747#issuecomment-2233662712, https://github.com/TileDB-Inc/TileDB-R/issues/730, https://github.com/TileDB-Inc/conda-forge-nightly-controller/issues/108#issuecomment-2233622013, https://github.com/TileDB-Inc/TileDB-MariaDB/issues/358

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Wednesday (2024-07-17) in run 9984394452

github-actions[bot] commented 1 month ago

The centralized nightlies job failed on Thursday (2024-07-18) in run 10001437480

github-actions[bot] commented 4 weeks ago

The centralized nightlies job failed on Friday (2024-07-19) in run 10016749877

github-actions[bot] commented 4 weeks ago

The centralized nightlies job failed on Saturday (2024-07-20) in run 10024869559

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Sunday (2024-07-21) in run 10033500413

jdblischak commented 3 weeks ago

TileDB-Py and libtiledbvcf continue to fail due to the use of deprecated APIs. The tests for tiledbsoma-r sometimes pass and sometimes segfault. Unclear if this is spurious or not. Will have to keep monitoring

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Monday (2024-07-22) in run 10051518578

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Tuesday (2024-07-23) in run 10069345493

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Wednesday (2024-07-24) in run 10087006106

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Thursday (2024-07-25) in run 10104559745

jdblischak commented 3 weeks ago

The TileDB-Py error about tiledb_array_delete_fragments is gone (as expected since https://github.com/TileDB-Inc/TileDB-Py/pull/2009 was merged).

However, there are still errors that are failing the build:

[12/33] Building CXX object tiledb/cc/CMakeFiles/cc.dir/array.cc.o
FAILED: tiledb/cc/CMakeFiles/cc.dir/array.cc.o
/usr/bin/g++ -Dcc_EXPORTS -isystem /opt/hostedtoolcache/Python/3.11.9/x64/include/python3.11 -isystem /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include -isystem /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/include -O3 -DNDEBUG -std=gnu++20 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT tiledb/cc/CMakeFiles/cc.dir/array.cc.o -MF tiledb/cc/CMakeFiles/cc.dir/array.cc.o.d -o tiledb/cc/CMakeFiles/cc.dir/array.cc.o -c /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc
/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc: In function ‘void libtiledbcpp::init_array(pybind11::module&)’:
/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc:100:76: error: no match for call to ‘(const pybind11::detail::overload_cast_impl<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const tiledb::ArraySchema&, tiledb_encryption_type_t, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>) (void (*)(const string&, const tiledb::ArraySchema&))’
   99 |            py::overload_cast<const std::string &, const ArraySchema &,
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  100 |                              tiledb_encryption_type_t, const std::string &>(
      |                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
  101 |                &Array::create))
      |                ~~~~~~~~~~~~~~~
In file included from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/attr.h:13,
                 from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/detail/class.h:12,
                 from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                 from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/numpy.h:12,
                 from /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/common.h:4,
                 from /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc:4:

The error appears related to pybind11. @teo-tsirpanis do you have any ideas what could be causing this?

teo-tsirpanis commented 3 weeks ago

Apparently one API was missed in TileDB-Inc/TileDB-Py#1958, c.c. @kounelisagis.

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Friday (2024-07-26) in run 10120050775

teo-tsirpanis commented 3 weeks ago

Oh my goodness, another one? 😩

jdblischak commented 3 weeks ago

Do we have to remove all instances of tiledb_encryption_type_t? If yes, I see 3 more instances in tiledb/cc/array.cc that remain after https://github.com/TileDB-Inc/TileDB-Py/pull/2021 was merged:

https://github.com/TileDB-Inc/TileDB-Py/blob/e29bf7301c44df9c8086c83a66deaf3c9089e12d/tiledb/cc/array.cc#L124 https://github.com/TileDB-Inc/TileDB-Py/blob/e29bf7301c44df9c8086c83a66deaf3c9089e12d/tiledb/cc/array.cc#L40 https://github.com/TileDB-Inc/TileDB-Py/blob/e29bf7301c44df9c8086c83a66deaf3c9089e12d/tiledb/cc/array.cc#L45

cc: @kounelisagis

github-actions[bot] commented 3 weeks ago

The centralized nightlies job failed on Saturday (2024-07-27) in run 10128189890

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Sunday (2024-07-28) in run 10136948012

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Monday (2024-07-29) in run 10154979104

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Tuesday (2024-07-30) in run 10172798362

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Wednesday (2024-07-31) in run 10190812724

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Thursday (2024-08-01) in run 10208863772

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Friday (2024-08-02) in run 10224480159

github-actions[bot] commented 2 weeks ago

The centralized nightlies job failed on Saturday (2024-08-03) in run 10232732546

github-actions[bot] commented 1 week ago

The centralized nightlies job failed on Sunday (2024-08-04) in run 10241445902

kounelisagis commented 1 week ago

For Python, should we do something similar to this https://github.com/TileDB-Inc/conda-forge-nightly-controller/pull/122 @jdblischak?

github-actions[bot] commented 1 week ago

The centralized nightlies job failed on Monday (2024-08-05) in run 10259108970

github-actions[bot] commented 1 week ago

The centralized nightlies job failed on Tuesday (2024-08-06) in run 10276973841

github-actions[bot] commented 1 week ago

The centralized nightlies job failed on Wednesday (2024-08-07) in run 10294780430

jdblischak commented 1 week ago

It's been a bit of a roller coaster ride, but things are settling down.

3 days ago there were 4 failings builds:

image

But last night only TileDB-Py failed. As @kounelisagis suggested above (https://github.com/jdblischak/centralized-tiledb-nightlies/issues/14#issuecomment-2268942904), I'll try addressing this with -Cskbuild.cmake.define.TILEDB_REMOVE_DEPRECATIONS=OFF.

The libtiledbvcf build was fixed by https://github.com/TileDB-Inc/TileDB-VCF/pull/753 and https://github.com/TileDB-Inc/TileDB-VCF/pull/755

Unclear why TileDB-Go is passing now. 3 days ago it was clearly failing due to using deprecated APIs (see test failure below). However, while there have been recent commits both to TileDB-Go and TileDB, none appear related to these particular deprecated APIs.

# github.com/TileDB-Inc/TileDB-Go [github.com/TileDB-Inc/TileDB-Go.test]
cgo-gcc-prolog: In function ‘_cgo_b6efb34e80b0_Cfunc_tiledb_enumeration_dump’:
cgo-gcc-prolog:252:2: warning: ‘tiledb_enumeration_dump’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb_experimental.h:46,
                 from ./enumeration_experimental.go:5:
./install-libtiledb/include/tiledb/api/c_api/enumeration/enumeration_api_experimental.h:314:40: note: declared here
  314 | TILEDB_DEPRECATED_EXPORT capi_return_t tiledb_enumeration_dump(
      |                                        ^~~~~~~~~~~~~~~~~~~~~~~

Also unclear why TileDB-R is passing now. Maybe the previous error was spurious. There have been no recent upstream commits.

g++ -std=gnu++17 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o tiledb.so RcppExports.o arrowio.o batched.o column_buffer.o deprecation.o durations.o libtiledb.o nanoarrow.o nullable.o shmem.o virtualfile.o -L/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib -ltiledb -Wl,-rpath,/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib -L/usr/lib/R/lib -lR
/usr/bin/ld:/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib/libtiledb.so: file format not recognized; treating as linker script
/usr/bin/ld:/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib/libtiledb.so:0: syntax error
collect2: error: ld returned 1 exit status
make: *** [/usr/share/R/share/make/shlib.mk:10: tiledb.so] Error 1
ERROR: compilation failed for package ‘tiledb’
* removing ‘/usr/local/lib/R/site-library/tiledb’
jdblischak commented 1 week ago

TileDB-Py build fixed by ec5ce1f3d9cd1d01f799a7d0f1faa76184d2d73d (build). Thanks @kounelisagis!

xref: https://github.com/TileDB-Inc/TileDB-Py/pull/2023, https://github.com/TileDB-Inc/conda-forge-nightly-controller/pull/122