[CI] Nightly integration test failure

[uenoku]

https://github.com/llvm/circt/actions/runs/9633232858/job/26567402157

: && /usr/lib/ccache/clang++ -fPIC -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-compat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -g  -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--gdb-index -Wl,--color-diagnostics -shared -Wl,-soname,libCIRCTScheduling.so -o lib/libCIRCTScheduling.so tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/ASAPScheduler.cpp.o tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/ChainingSupport.cpp.o tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/Problems.cpp.o tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/SimplexSchedulers.cpp.o tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/Utilities.cpp.o tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/LPSchedulers.cpp.o tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/CPSATSchedulers.cpp.o  -Wl,-rpath,/__w/circt/circt/build/lib:/usr/local/lib:  lib/libMLIRFuncDialect.so  /usr/lib/libortools.so.9.5.9999  lib/libMLIRCallInterfaces.so  lib/libMLIRControlFlowInterfaces.so  lib/libMLIRFunctionInterfaces.so  lib/libMLIRInferTypeOpInterface.so  lib/libMLIRSideEffectInterfaces.so  lib/libMLIRIR.so  lib/libMLIRSupport.so  lib/libLLVMSupport.so  -ldl  /usr/local/lib/libabsl_flags_parse.a  /usr/local/lib/libabsl_flags_usage.a  /usr/local/lib/libabsl_flags_usage_internal.a  /usr/local/lib/libabsl_flags.a  /usr/local/lib/libabsl_flags_internal.a  /usr/local/lib/libabsl_flags_marshalling.a  /usr/local/lib/libabsl_flags_reflection.a  /usr/local/lib/libabsl_flags_config.a  /usr/local/lib/libabsl_flags_private_handle_accessor.a  /usr/local/lib/libabsl_flags_commandlineflag.a  /usr/local/lib/libabsl_flags_commandlineflag_internal.a  /usr/local/lib/libabsl_flags_program_name.a  /usr/local/lib/libabsl_random_distributions.a  /usr/local/lib/libabsl_random_seed_sequences.a  /usr/local/lib/libabsl_random_internal_pool_urbg.a  /usr/local/lib/libabsl_random_internal_randen.a  /usr/local/lib/libabsl_random_internal_randen_hwaes.a  /usr/local/lib/libabsl_random_internal_randen_hwaes_impl.a  /usr/local/lib/libabsl_random_internal_randen_slow.a  /usr/local/lib/libabsl_random_internal_platform.a  /usr/local/lib/libabsl_random_internal_seed_material.a  /usr/local/lib/libabsl_random_seed_gen_exception.a  /usr/local/lib/libabsl_raw_hash_set.a  /usr/local/lib/libabsl_hashtablez_sampler.a  /usr/local/lib/libabsl_hash.a  /usr/local/lib/libabsl_city.a  /usr/local/lib/libabsl_low_level_hash.a  /usr/local/lib/libabsl_leak_check.a  /usr/local/lib/libabsl_statusor.a  /usr/local/lib/libabsl_status.a  /usr/local/lib/libabsl_cord.a  /usr/local/lib/libabsl_cordz_info.a  /usr/local/lib/libabsl_cord_internal.a  /usr/local/lib/libabsl_cordz_functions.a  /usr/local/lib/libabsl_exponential_biased.a  /usr/local/lib/libabsl_cordz_handle.a  /usr/local/lib/libabsl_crc_cord_state.a  /usr/local/lib/libabsl_crc32c.a  /usr/local/lib/libabsl_crc_internal.a  /usr/local/lib/libabsl_crc_cpu_detect.a  /usr/local/lib/libabsl_bad_optional_access.a  /usr/local/lib/libabsl_strerror.a  /usr/local/lib/libabsl_bad_variant_access.a  /usr/local/lib/libabsl_str_format_internal.a  /usr/local/lib/libabsl_synchronization.a  /usr/local/lib/libabsl_stacktrace.a  /usr/local/lib/libabsl_symbolize.a  /usr/local/lib/libabsl_debugging_internal.a  /usr/local/lib/libabsl_demangle_internal.a  /usr/local/lib/libabsl_graphcycles_internal.a  /usr/local/lib/libabsl_malloc_internal.a  /usr/local/lib/libabsl_time.a  /usr/local/lib/libabsl_strings.a  /usr/local/lib/libabsl_strings_internal.a  /usr/local/lib/libabsl_base.a  /usr/local/lib/libabsl_spinlock_wait.a  -lrt  /usr/local/lib/libabsl_throw_delegate.a  /usr/local/lib/libabsl_int128.a  /usr/local/lib/libabsl_civil_time.a  /usr/local/lib/libabsl_time_zone.a  /usr/local/lib/libabsl_bad_any_cast_impl.a  /usr/local/lib/libabsl_raw_logging_internal.a  /usr/local/lib/libabsl_log_severity.a  /usr/local/lib/libprotobuf.a  -pthread  /usr/local/lib/libre2.a  /usr/lib/libCbcSolver.a  /usr/lib/libOsiCbc.a  /usr/lib/libCbc.a  /usr/lib/libCgl.a  /usr/lib/libClpSolver.a  /usr/lib/libOsiClp.a  /usr/lib/libClp.a  /usr/lib/libOsi.a  /usr/lib/libCoinUtils.a  /usr/local/lib/libz.so  -lpthread  -Wl,-rpath-link,/__w/circt/circt/build/lib && :
ld.lld: error: undefined symbol: operations_research::MPSolver::ParseSolverType(absl::lts_20230125::string_view, operations_research::MPSolver::OptimizationProblemType*)
>>> referenced by LPSchedulers.cpp:30 (/__w/circt/circt/lib/Scheduling/LPSchedulers.cpp:30)
>>>               tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/LPSchedulers.cpp.o:(circt::scheduling::scheduleLP(circt::scheduling::Problem&, mlir::Operation*))
>>> referenced by LPSchedulers.cpp:87 (/__w/circt/circt/lib/Scheduling/LPSchedulers.cpp:87)
>>>               tools/circt/lib/Scheduling/CMakeFiles/obj.CIRCTScheduling.dir/LPSchedulers.cpp.o:(circt::scheduling::scheduleLP(circt::scheduling::CyclicProblem&, mlir::Operation*))
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Probably due to docker image bump v13.1 -> v15 in https://github.com/llvm/circt/commit/88eeb265f479ab9cd7800d9859dd623f4d90c891?

[mingzheTerapines]

Thanks for reminding.

[teqdruid]

This smells kinda like a caching issue... I'm gonna try clearing the relevant caches.

[teqdruid]

Nope. Clearing short integration tests cache didn't work. Same issue. I'm gonna revert the clang version change in the image.

[teqdruid]

Oh shit... the clang change was for the regular build image, not the integration test image.

[teqdruid]

Here's the 13.1 -> 15.0 image build diff: https://github.com/circt/images/compare/v13.1...v15.0#diff-6a1298b7c0a74ef094b033382b6e61d38a5213fcff54392b62b4d7880d5b6934 . I don't see any changes in there (in the integration test image) which could cause this issue...

[fzi-hielscher]

I think absl::lts_20230125::string_view is the problem here. That is not the abseil version or-tools builds with. Its the one shipped by gRPC. I think the CIRCT scheduling lib is pulling in the wrong header during compilation, so the signature doesn't line up with the or-tools library.

[teqdruid]

Yeah, that's what I was just investigating.

[teqdruid]

An interesting question is why the PR which contained the image change passed the short integration tests build but now the short integration tests build is failing.

[teqdruid]

I feel like I'm pretty close to a solution here: build Abseil on its own (and install it), then have gRPC and or-tools use the system-installed version. I'm just working through some "details" now but probably won't have time to finish today.

[teqdruid]

My current work-in-progress is here: https://github.com/circt/images/tree/teqdruid/fix-int-image

[teqdruid]

Well, I wasn't even close. Why is this so damn difficult to get right? or-tools seems very sensitive to the way Abseil is compiled.

[teqdruid]

I gave up, reverted the image, and disabled the ESI runtime build #7236. That PR should fix the nightlies and short integration test.

[teqdruid]

Last night's nightlies didn't fail last night. Closing this issue.