search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.28k stars 1.26k forks source link

CI failure: "Missing digest" #21255

Open BetsyMcPhail opened 6 months ago

BetsyMcPhail commented 6 months ago

On 4/2/24, buildcops began noticing CI jobs failing with errors like:

[10:15:14 AM] ERROR: /media/ephemeral0/ubuntu/workspace/linux-jammy-gcc-bazel-continuous-debug/src/math/BUILD.bazel:427:20: Compiling math/test/discrete_algebraic_riccati_equation_test.cc failed: unable to finalize action: Missing digest: 6f5ad5446e49451a55dddee73681d3f44103b4aea9c85b308a9761effe9e3693/34778 for bazel-out/k8-dbg/bin/math/_objs/discrete_algebraic_riccati_equation_test/discrete_algebraic_riccati_equation_test.pic.d

The build cache key was incremented in https://github.com/RobotLocomotion/drake-ci/pull/276 as an attempted solution, but the failure continues to sporadically appear.

See slack conversation for initial discussion.

BetsyMcPhail commented 6 months ago

4/15: https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/495/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/497/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-debug/179/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/495/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/497/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-gcc-bazel-continuous-debug/1671/

SeanCurtis-TRI commented 5 months ago

4/18

https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/514/

xuchenhan-tri commented 5 months ago

4/24 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/537/

BetsyMcPhail commented 4 months ago

5/20 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-debug/283/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-debug/284/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/607/

mwoehlke-kitware commented 4 months ago

5/28 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/622/

BetsyMcPhail commented 4 months ago

6/4 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-debug/320/

BetsyMcPhail commented 3 months ago

6/24 https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-jammy-gcc-bazel-continuous-debug/1833

xuchenhan-tri commented 3 months ago

6/25 https://drake-jenkins.csail.mit.edu/view/Production/job/linux-jammy-clang-bazel-continuous-debug/351 https://drake-jenkins.csail.mit.edu/view/Production/job/linux-jammy-clang-bazel-continuous-debug/352 https://drake-jenkins.csail.mit.edu/view/Production/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/674

BetsyMcPhail commented 2 months ago

8/5 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/787/

mwoehlke-kitware commented 2 months ago

8/6 https://drake-jenkins.csail.mit.edu/view/Production/job/linux-jammy-clang-bazel-continuous-everything-address-sanitizer/796/

BetsyMcPhail commented 1 month ago

9/10 https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/867/ https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/868/

nicolecheetham commented 4 days ago

10/9 https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/960/ https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-jammy-clang-bazel-continuous-leak-sanitizer/962/ https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/job/linux-jammy-gcc-bazel-continuous-debug/2136/