search

openxla / xla

A machine learning compiler for GPUs, CPUs, and ML accelerators
Apache License 2.0
2.69k stars 434 forks source link

no such target @local_config_nccl//:nccl_headers #17326

Closed apivovarov closed 1 month ago

apivovarov commented 1 month ago

Error:

bazel build //xla/tsl/cuda:nccl

ERROR: /home/ubuntu/workspace/xla/xla/tsl/cuda/BUILD.bazel:336:11: no such target '@local_config_nccl//:nccl_headers': target 'nccl_headers' not declared in package '' defined by /home/ubuntu/.cache/bazel/_bazel_ubuntu/0176aac02e77df836cc7203737c26784/external/local_config_nccl/BUILD (Tip: use `query "@local_config_nccl//:*"` to see all the targets in that package) and referenced by '//xla/tsl/cuda:nccl'
ERROR: Analysis of target '//xla/tsl/cuda:nccl' failed; build aborted: Analysis failed

bazel build //xla/...

ERROR: /home/ubuntu/workspace/xla/xla/tsl/cuda/BUILD.bazel:336:11: no such target '@local_config_nccl//:nccl_headers': target 'nccl_headers' not declared in package '' defined by /home/ubuntu/.cache/bazel/_bazel_ubuntu/0176aac02e77df836cc7203737c26784/external/local_config_nccl/BUILD (Tip: use `query "@local_config_nccl//:*"` to see all the targets in that package) and referenced by '//xla/tsl/cuda:nccl'
ERROR: Analysis of target '//xla/tsl/cuda:nccl' failed; build aborted: Analysis failed

Bazel Query:

bazel query @local_config_nccl//:*

@local_config_nccl//:BUILD
@local_config_nccl//:LICENSE
@local_config_nccl//:nccl
@local_config_nccl//:nccl_config
@local_config_nccl//:nccl_config.h
apivovarov commented 1 month ago

Hi David, could you help with triaging and assigning this issue? It blocks //xla/tsl/.. testing on GPU.

@ddunl

ddunl commented 1 month ago

Ah I see - I'm wondering if there's some interaction here with hermetic cuda.

sergachev commented 1 month ago

Related (following https://openxla.org/xla/lsp):

bazel aquery "mnemonic(CppCompile, //xla/...)" --output=jsonproto
...
xla/xla/tsl/cuda/BUILD.bazel:336:11: no such target '@local_config_nccl//:nccl_headers': target 'nccl_headers' not declared in package '' defined by ...
sergachev commented 1 month ago

And for me this fixes the problem, at least in a setup not using NCCL: https://github.com/openxla/xla/pull/17369

ybaturina commented 1 month ago

I've submitted the fix in https://github.com/openxla/xla/pull/17377. Please let me know if it works for you.

sergachev commented 1 month ago

I've submitted the fix in https://github.com/openxla/xla/pull/17377. Please let me know if it works for you.

Works for me.

apivovarov commented 1 month ago

bazel build //xla/tsl/cuda:nccl - works fine now. Thank you!