The nightly build job failed on Wednesday (2024-02-07)

[github-actions[bot]]

The nightly build job failed on Wednesday (2024-02-07) in run 7824279607

[jdblischak]

The nightly osx-libtiledb-dev build failed here and on my fork. They both failed at the same step when running ./build-libtiledbvcf/libtiledbvcf/test/tiledb_vcf_unit. However the error messages are different:

./build-libtiledbvcf/libtiledbvcf/test/tiledb_vcf_unit

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tiledb_vcf_unit is a Catch v2.13.8 host application.
Run with -? for options

-------------------------------------------------------------------------------
TileDB-VCF: Test export
-------------------------------------------------------------------------------
/Users/runner/work/TileDB-VCF/TileDB-VCF/TileDB-VCF/libtiledbvcf/test/src/unit-vcf-export.cc:192
...............................................................................

/Users/runner/work/TileDB-VCF/TileDB-VCF/TileDB-VCF/libtiledbvcf/test/src/unit-vcf-export.cc:192: FAILED:
  {Unknown expression after the reported line}
due to unexpected exception with message:
  Error loading metadata; 'version' field has invalid value.

===============================================================================
test cases:   78 |   77 passed | 1 failed
assertions: 6784 | 6783 passed | 1 failed

./build-libtiledbvcf/libtiledbvcf/test/tiledb_vcf_unit

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
tiledb_vcf_unit is a Catch v2.13.8 host application.
Run with -? for options

-------------------------------------------------------------------------------
TileDB-VCF: Test Resume Contig Merge
-------------------------------------------------------------------------------
/Users/runner/work/TileDB-VCF/TileDB-VCF/TileDB-VCF/libtiledbvcf/test/src/unit-vcf-store.cc:648
...............................................................................

/Users/runner/work/TileDB-VCF/TileDB-VCF/TileDB-VCF/libtiledbvcf/test/src/unit-vcf-store.cc:681: FAILED:
  REQUIRE( fragmentInfo.fragment_num() == 38 )
with expansion:
  [33](https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/7824279607/job/21346593253#step:9:34) == [38](https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/7824279607/job/21346593253#step:9:39)

===============================================================================
test cases:   78 |   77 passed | 1 failed
assertions: 6820 | 6819 passed | 1 failed

[ihnorton]

@awenocur please investigate.

[github-actions[bot]]

The nightly build job failed on Saturday (2024-02-10) in run 7859247915

[github-actions[bot]]

The nightly build job failed on Monday (2024-02-12) in run 7881120014

[github-actions[bot]]

The nightly build job failed on Tuesday (2024-02-13) in run 7895881600

[jdblischak]

The osx-specific error above is now masked by the known libboost error (https://github.com/TileDB-Inc/conda-forge-nightly-controller/issues/49), which is now causing both the linux and osx builds against dev libtiledb to fail

[jdblischak]

More details. The libtiledb nightly build fails because this repo is using TILEDB_VCPKG=OFF for its nightly builds

https://github.com/TileDB-Inc/TileDB-VCF/blob/e83a8fcdb58161ef7916d0fab4d033fe77592690/ci/nightly/build-libtiledb.sh#L11

Whether or not to add boost as an external project is being discussed in https://github.com/TileDB-Inc/TileDB/pull/4723 and https://github.com/TileDB-Inc/TileDB/pull/4724

[github-actions[bot]]

The nightly build job failed on Wednesday (2024-02-14) in run 7910509791

[jdblischak]

The nightly build passed last night

https://github.com/TileDB-Inc/TileDB-VCF/actions/runs/7925426405

The boost errors are gone because this dependency was removed upstream in https://github.com/TileDB-Inc/TileDB/pull/4731

I don't know how the osx-specific test failure was fixed

@awenocur please close this issue

[ihnorton]

I think we ended up with an earlier arrow version (9) in that run, which avoided the InitAPI issue we had previously seen on osx (sc-38823).

[jdblischak]

I think we ended up with an earlier arrow version (9) in that run,

The passing run installed pyarrow 11 because the pyarrow version is hard-coded for the nightly builds:

https://github.com/TileDB-Inc/TileDB-VCF/blob/ab468471804d3f5ccf1dec5d278698218a51db72/.github/workflows/nightly.yml#L104

https://github.com/TileDB-Inc/TileDB-VCF/blob/ab468471804d3f5ccf1dec5d278698218a51db72/ci/nightly/requirements.txt#L3

which avoided the InitAPI issue we had previously seen on osx (sc-38823).

AFAICT the AWS error on osx only occurs in our conda feedstock builds. The original error from two weeks ago that opened this Issue was due to a test failure when running ./build-libtiledbvcf/libtiledbvcf/test/tiledb_vcf_unit, ie before pyarrow was even installed. Whatever was causing that error is gone now.