[Bug]: cannot use `grpc` and `googleapis` with address sanitizer

[coryan]

What happened?

Running a test with address sanitizer which includes gRPC and googleapis results in ODR violations. It is unclear if this is a problem with the BCR modules for googleapis, for gRPC, or the problems are in the underlying code.

Version

Development (host) and target OS/architectures:

Linux, targeting Linux on x86.

Output of bazel --version:

bazel 7.2.1

Version of relevant rules from the WORKSPACE or MODULE.bazel file:

Language(s) and/or frameworks involved:

C++

How to reproduce

# Extract the attached tarball:
tar -zxvf bzlmod-odr.tar.gz

# If docker is available, this will install a toolchain and test the code:
docker buildx build bzlmod-odr

# If you prefer to install things manually:
cd bzlmod-odr
bazel test --test_output=errors --config asan :hello

Any other information?

bzlmod-odr.tar.gz

[coryan]

Maybe this is the root cause: https://github.com/grpc/grpc/issues/36983

[keith]

can you post the specific error? i think there might be an error here where grpc is vendoring googleapis and googleapis is coming from the BCR, it might be that one, which we should fix if so

[coryan]

Starting local Bazel server and connecting to it...
INFO: Analyzed target //:hello (227 packages loaded, 6575 targets configured).
FAIL: //:hello (see /usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/testlogs/hello/test.log)
INFO: From Testing //:hello:
==================== Test output for //:hello:
=================================================================
==12==ERROR: AddressSanitizer: odr-violation (0x7f93dcd03c80):
  [1] size=40 'google_rpc_status_proto_upb_file_layout' bazel-out/k8-fastbuild/bin/external/grpc~~grpc_repo_deps_ext~com_google_googleapis/google/rpc/status.upb_minitable.c:42:25 in /usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/bin/_solib_k8/libexternal_Sgrpc~~grpc_Urepo_Udeps_Uext~com_Ugoogle_Ugoogleapis_Sgoogle_Srpc_Slibstatus_Uproto.upb_Uminitable.so
  [2] size=40 'google_rpc_status_proto_upb_file_layout' bazel-out/k8-fastbuild/bin/external/googleapis~/google/rpc/status.upb_minitable.c:42:25 in /usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/bin/_solib_k8/libexternal_Sgoogleapis~_Sgoogle_Srpc_Slibstatus_Uproto.upb_Uminitable.so
These globals were registered at these points:
  [1]:
    #0 0x7f93df64f4d7 in __asan_register_globals ../../../../src/libsanitizer/asan/asan_globals.cpp:350
    #1 0x7f93dcd01c4e in _sub_I_00099_1 (/usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/bin/_solib_k8/libexternal_Sgrpc~~grpc_Urepo_Udeps_Uext~com_Ugoogle_Ugoogleapis_Sgoogle_Srpc_Slibstatus_Uproto.upb_Uminitable.so+0xc4e) (BuildId: 12102707363b3b47486273ee53a688e50f0088c8)
    #2 0x7f93dfe00e3d in call_init elf/dl-init.c:74
    #3 0x7f93dfe00e3d in call_init elf/dl-init.c:26

  [2]:
    #0 0x7f93df64f4d7 in __asan_register_globals ../../../../src/libsanitizer/asan/asan_globals.cpp:350
    #1 0x7f93dbd81c2e in _sub_I_00099_1 (/usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/bin/_solib_k8/libexternal_Sgoogleapis~_Sgoogle_Srpc_Slibstatus_Uproto.upb_Uminitable.so+0xc2e) (BuildId: ff21305364be0c3f404489842dea6ae6d2852978)
    #2 0x7f93dfe00e3d in call_init elf/dl-init.c:74
    #3 0x7f93dfe00e3d in call_init elf/dl-init.c:26

==12==HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_odr_violation=0
SUMMARY: AddressSanitizer: odr-violation: global 'google_rpc_status_proto_upb_file_layout' at bazel-out/k8-fastbuild/bin/external/grpc~~grpc_repo_deps_ext~com_google_googleapis/google/rpc/status.upb_minitable.c:42:25 in /usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/bin/_solib_k8/libexternal_Sgrpc~~grpc_Urepo_Udeps_Uext~com_Ugoogle_Ugoogleapis_Sgoogle_Srpc_Slibstatus_Uproto.upb_Uminitable.so
==12==ABORTING
================================================================================
INFO: Found 1 test target...
Target //:hello up-to-date:
  /usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/bin/hello
INFO: Elapsed time: 8.590s, Critical Path: 1.65s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//:hello                                                                 FAILED in 1.0s
  /usr/local/google/home/coryan/.cache/bazel/_bazel_coryan/7086fe033be11da621e2536ef63e9c11/execroot/_main/bazel-out/k8-fastbuild/testlogs/hello/test.log

Executed 1 out of 1 test: 1 fails locally.

[keith]

yea this looks like the case i was thinking, there are 2 versions of googleapis in the build

[coryan]

yea this looks like the case i was thinking, there are 2 versions of googleapis in the build

Is this a problem with my test project? Meaning, should I fix the MODULE.bazel file or any of the other files in my repro? Or is the problem in BCR and/or the packages themselves?

[keith]

no it's a problem in the grpc MODULE.bazel, it should move from a non-bzlmod googleapis to the bcr version