googlecloudrobotics / core

Cloud Robotics Core: Kubernetes, Federation, App Management
Apache License 2.0
192 stars 61 forks source link

[Bazel CI] app_management_test is failing with Bazel@HEAD #447

Open sgowroji opened 17 hours ago

sgowroji commented 17 hours ago

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4142#01921d0e-ec6d-44af-a755-5c6640f93d9c

Platform: Ubuntu

Logs:

FAIL: //src/app_charts/base:app_management_test (Exit 1) (see /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/e68609a51962c4af9517a3c40321ba14/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/app_charts/base/app_management_test/test_attempts/attempt_1.log)
FAIL: //src/app_charts/base:app_management_test (Exit 1) (see /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/e68609a51962c4af9517a3c40321ba14/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/app_charts/base/app_management_test/test_attempts/attempt_2.log)
FAIL: //src/app_charts/base:app_management_test (Exit 1) (see /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/e68609a51962c4af9517a3c40321ba14/execroot/_main/bazel-out/k8-fastbuild/testlogs/src/app_charts/base/app_management_test/test.log)
FAILED: //src/app_charts/base:app_management_test (Summary)
Error: The command exited with status 1
error: The plugin docker command hook exited with status 1

Culprit:

Steps:

 git clone https://github.com/googlecloudrobotics/core
git reset c66169cd7e5d27849c2e777dba7e4b6949e87768  --hard
export USE_BAZEL_VERSION=b084956e23a6e50e35fe8886d0f3f7e05f718155
bazel test //... 

CC Greenteam @wyverald

drigz commented 14 hours ago

error is /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/e68609a51962c4af9517a3c40321ba14/sandbox/linux-sandbox/2734/execroot/_main/bazel-out/k8-fastbuild/bin/src/app_charts/base/app_management_test.runfiles/_main/external/_main~non_module_deps~kubernetes_helm/helm: No such file or directory

my guess is that this magic path is no longer correct after some update: https://github.com/googlecloudrobotics/core/blob/c66169cd7e5d27849c2e777dba7e4b6949e87768/src/app_charts/base/app_management_test.sh#L17

I guess there must be some better way for sh_test to find a runfile, something similar to a genrule's $(location :tgt), but I don't know what it is...

ensonic commented 14 hours ago

See https://github.com/googlecloudrobotics/core/issues/417#issuecomment-2363326574. There is a proposal, but its also quite involved as it seems. I've tried a simpler version on Friday (https://github.com/googlecloudrobotics/core/pull/445), but had to revert it as it failed on our ci :/