search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.35k stars 1.27k forks source link

Address sporadic kcov segmentation faults #20847

Open liangfok opened 10 months ago

liangfok commented 10 months ago

What happened?

The following failure occured in nightly production:

/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/_bazel_ubuntu/ca8cca4382e64e0e68b3df6a056070b5/execroot/drake/bazel-out/k8-dbg/bin/solvers/mosek_solver_internal_test.runfiles/drake/tools/dynamic_analysis/kcov: line 65: 75308 Segmentation fault      (core dumped) kcov "--include-path=/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/src" --verify --python-parse=python3 --exclude-pattern=third_party "--replace-src-path=/proc/self/cwd:/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/src" "/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/_bazel_ubuntu/ca8cca4382e64e0e68b3df6a056070b5/execroot/drake/bazel-out/k8-dbg/testlogs/solvers/mosek_solver_internal_test/test.outputs/kcov" solvers/mosek_solver_internal_test
Could not create "/media/ephemeral0/ubuntu/workspace/linux-jammy-clang-bazel-nightly-coverage/_bazel_ubuntu/ca8cca4382e64e0e68b3df6a056070b5/execroot/drake/bazel-out/k8-dbg/testlogs/solvers/mosek_solver_internal_test/test.outputs/outputs.zip": zip not found or failed

//solvers:mosek_solver_internal_test                                     FAILED in 3.0s

Links:

  1. https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/116/
  2. https://drake-cdash.csail.mit.edu/test/1343352056
liangfok commented 10 months ago

Closing since this is first occurrence.

SeanCurtis-TRI commented 8 months ago

We had another incidence today -- albeit in a different test.

https://drake-jenkins.csail.mit.edu/view/Weekly%20Production/job/linux-jammy-clang-bazel-weekly-everything-coverage/21/

BetsyMcPhail commented 8 months ago

Three more failures, all in different tests:

BetsyMcPhail commented 8 months ago

3/29 segfault in //multibody/contact_solvers:minimum_degree_ordering_test in https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/175/

BetsyMcPhail commented 7 months ago

4/1 segfault in //math:hopf_coordinate_test: linux-jammy-gcc-bazel-nightly-coverage

xuchenhan-tri commented 7 months ago

Reopening the issue per the buildcop playbook. I tried reproducing the segfault locally but wasn't able to do it with a handful of runs.

BetsyMcPhail commented 7 months ago

4/9 //multibody/fem:dirichlet_boundary_condition_test linux-jammy-clang-bazel-nightly-coverage

BetsyMcPhail commented 7 months ago

4/15: //multibody/parsing:package_map_remote_test https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/job/linux-jammy-clang-bazel-nightly-coverage/192/

BetsyMcPhail commented 6 months ago

5/20: //common:text_logging_test failed in linux-jammy-gcc-bazel-weekly-everything-coverage

BetsyMcPhail commented 6 months ago

5/21 //common:dummy_value_test failed in linux-jammy-clang-bazel-nightly-coverage

williamjallen commented 2 months ago

9/19 //common:extract_double_test failed in linux-jammy-clang-bazel-nightly-coverage

williamjallen commented 2 months ago

9/24 //manipulation/kuka_iiwa:iiwa_status_receiver_test failed in https://drake-jenkins.csail.mit.edu/job/linux-jammy-clang-bazel-nightly-coverage/355/consoleFull

BetsyMcPhail commented 6 days ago

11/19 - linux-jammy-clang-bazel-nightly-coverage failed with a segfault in //systems/primitives:vector_log_test