search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.71k stars 3.98k forks source link

bazel coverage doesn't produce coverage data for cuda config for XLA. #22709

Open hmonishN opened 3 weeks ago

hmonishN commented 3 weeks ago

Description of the bug:

bazel coverage doesn't seem to produce coverage data for cuda configuration for XLA tests.

gives output: WARNING: There was no coverage found Also, the coverage report is an empty file.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Steps for cuda config : (Can either use jax latest container or make sure cuda tk 12.3 is installed with cudnn 8.7.6 or higher is insatlled) git clone cd xla export TF_NEED_CUDA=1 export TF_CUDA_COMPUTE_CAPABILITIES="compute_80" bazel coverage --config cuda --instrument_test_targets --experimental_cc_coverage --combined_report=lcov //xla/service/gpu:command_buffer_scheduling_test

Which operating system are you running Bazel on?

Ubuntu 22.04.4 LTS

What is the output of bazel info release?

release 6.1.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

hmonishN commented 3 weeks ago

CC @dariuskia

sgowroji commented 3 weeks ago

Hi @hmonishN, Could you please provide sample code to reproduce this issue with complete steps?

hmonishN commented 3 weeks ago

Here are the steps to reproduce the issue on Ubuntu 22.04.4 LTS nvidia gpu (A100 or newer version) machine: sudo docker run --gpus all --security-opt seccomp=unconfined ghcr.io/nvidia/jax:latest cd /opt/xla export TF_NEED_CUDA=1 export TF_CUDA_COMPUTE_CAPABILITIES="compute_80" /opt/jax/build/bazel-6.1.2-linux-x86_64 coverage --config cuda --instrument_test_targets --experimental_cc_coverage --combined_report=lcov //xla/service/gpu:command_buffer_scheduling_test

hmonishN commented 3 weeks ago

The JAX container is an opensource container which comes with xla and cuda libs and toolkit installed in it with bazel 6.12 linux version. The definition of the xla test is defined here: https://github.com/openxla/xla/blob/main/xla/service/gpu/BUILD#L3261

hmonishN commented 2 weeks ago

Hi @sgowroji, any update on this? Is there a workaround that we can implement to make it work?