search

google / jax

Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
http://jax.readthedocs.io/
Apache License 2.0
29.94k stars 2.74k forks source link

Error compile: make_batch_pointers.cu.cc failed: undeclared inclusion(s) #23573

Closed johnnynunez closed 4 days ago

johnnynunez commented 6 days ago

Description

ERROR: /home/johnny/Projects/jax/jaxlib/cuda/BUILD:75:13: Compiling jaxlib/gpu/make_batch_pointers.cu.cc failed: undeclared inclusion(s) in rule '//jaxlib/cuda:cuda_make_batch_pointers': this rule is missing dependency declarations for the following files included by 'jaxlib/gpu/make_batch_pointers.cu.cc':

python3 build/build.py --enable_cuda --enable_nccl=False \
        --cuda_compute_capabilities="sm_87" \
        --cuda_version=12.2.0 --cudnn_version=8.9.4.25 \
        --bazel_options=--repo_env=LOCAL_CUDA_PATH="/usr/local/cuda-12.2" \
        --bazel_options=--repo_env=LOCAL_CUDNN_PATH="/opt/nvidia/cudnn/"

System info (python version, jaxlib version, accelerator, etc.)

python 3.10 jetson agx orin cuda 12.2

justinjfu commented 6 days ago

cc @dfm

hawkinsp commented 6 days ago

Can you share the complete error? That could either be misconfiguration of the compiler toolchain/libraries or a bug in JAX, and it's hard to say which.

dfm commented 5 days ago

@johnnynunez — like @hawkinsp says, can you post the complete error here so that we can work out what's going on?

johnnynunez commented 5 days ago 
(py310) johnny@johnny-jetson:~/Projects/jax$ python3 build/build.py --enable_cuda --enable_nccl=False --verbose=True         --cuda_compute_capabilities="sm_87"         --cuda_version=12.2.0 --cudnn_version=8.9.4.25         --bazel_options=--repo_env=LOCAL_CUDA_PATH="/usr/local/cuda-12.2"         --bazel_options=--repo_env=LOCAL_CUDNN_PATH="/opt/nvidia/cudnn/"

     _   _  __  __
    | | / \ \ \/ /
 _  | |/ _ \ \  /
| |_| / ___ \/  \
 \___/_/   \/_/\_\

INFO:__main__:shell(): ['./bazel-6.5.0-linux-arm64', '--version']
Bazel binary path: ./bazel-6.5.0-linux-arm64
Bazel version: 6.5.0
Python binary path: /home/johnny/Projects/py310/bin/python3
INFO:__main__:shell(): ['/home/johnny/Projects/py310/bin/python3', '-c', 'import sys; print("{}.{}".format(sys.version_info[0], sys.version_info[1]))']
Python version: 3.10
Use clang: yes
clang path: /usr/lib/llvm-14/bin/clang
MKL-DNN enabled: yes
Target CPU: aarch64
Target CPU features: release
CUDA enabled: yes
CUDA compute capabilities: sm_87
CUDA version: 12.2.0
CUDNN version: 8.9.4.25
NCCL enabled: no
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
./bazel-6.5.0-linux-arm64 run --verbose_failures=true --repo_env=LOCAL_CUDA_PATH=/usr/local/cuda-12.2 --repo_env=LOCAL_CUDNN_PATH=/opt/nvidia/cudnn/ //jaxlib/tools:build_wheel -- --output_path=/home/johnny/Projects/jax/dist --jaxlib_git_hash=1594d2f30fdbfebf693aba4a2b264e4a3e52acc6 --cpu=aarch64
INFO:__main__:shell(): ['./bazel-6.5.0-linux-arm64', 'run', '--verbose_failures=true', '--repo_env=LOCAL_CUDA_PATH=/usr/local/cuda-12.2', '--repo_env=LOCAL_CUDNN_PATH=/opt/nvidia/cudnn/', '//jaxlib/tools:build_wheel', '--', '--output_path=/home/johnny/Projects/jax/dist', '--jaxlib_git_hash=1594d2f30fdbfebf693aba4a2b264e4a3e52acc6', '--cpu=aarch64']
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/johnny/Projects/jax/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /home/johnny/Projects/jax/.bazelrc:
  Inherited 'build' options: --nocheck_visibility --apple_platform_type=macos --macos_minimum_os=10.14 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --experimental_cc_shared_library --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true --define=tsl_link_protobuf=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /home/johnny/Projects/jax/.jax_configure.bazelrc:
  Inherited 'build' options: --strategy=Genrule=standalone --action_env CLANG_COMPILER_PATH=/usr/lib/llvm-14/bin/clang --repo_env CC=/usr/lib/llvm-14/bin/clang --repo_env BAZEL_COMPILER=/usr/lib/llvm-14/bin/clang --copt=-Wno-error=unused-command-line-argument --config=mkl_open_source_only --config=cuda --config=nonccl --config=nvcc_clang --action_env=CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-14/bin/clang --repo_env HERMETIC_CUDA_VERSION=12.2.0 --repo_env HERMETIC_CUDNN_VERSION=8.9.4.25 --repo_env HERMETIC_PYTHON_VERSION=3.10
INFO: Found applicable config definition build:short_logs in file /home/johnny/Projects/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file /home/johnny/Projects/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:cuda in file /home/johnny/Projects/jax/.bazelrc: --repo_env TF_NEED_CUDA=1 --repo_env TF_NCCL_USE_STUB=1 --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_80,compute_90 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --@xla//xla/python:jax_cuda_pip_rpaths=true --repo_env=HERMETIC_CUDA_VERSION=12.3.2 --repo_env=HERMETIC_CUDNN_VERSION=9.1.1 --linkopt=-Wl,--disable-new-dtags
INFO: Found applicable config definition build:cuda in file /home/johnny/Projects/jax/.jax_configure.bazelrc: --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_87
INFO: Found applicable config definition build:nonccl in file /home/johnny/Projects/jax/.bazelrc: --define=no_nccl_support=true
INFO: Found applicable config definition build:nvcc_clang in file /home/johnny/Projects/jax/.bazelrc: --config=cuda --config=cuda_clang --action_env=TF_NVCC_CLANG=1 --@local_config_cuda//:cuda_compiler=nvcc
INFO: Found applicable config definition build:cuda in file /home/johnny/Projects/jax/.bazelrc: --repo_env TF_NEED_CUDA=1 --repo_env TF_NCCL_USE_STUB=1 --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_80,compute_90 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --@xla//xla/python:jax_cuda_pip_rpaths=true --repo_env=HERMETIC_CUDA_VERSION=12.3.2 --repo_env=HERMETIC_CUDNN_VERSION=9.1.1 --linkopt=-Wl,--disable-new-dtags
INFO: Found applicable config definition build:cuda in file /home/johnny/Projects/jax/.jax_configure.bazelrc: --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=sm_87
INFO: Found applicable config definition build:cuda_clang in file /home/johnny/Projects/jax/.bazelrc: --@local_config_cuda//:cuda_compiler=clang --action_env=CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-18/bin/clang --copt=-Wno-gnu-offsetof-extensions --copt=-Qunused-arguments
INFO: Found applicable config definition build:linux in file /home/johnny/Projects/jax/.bazelrc: --config=posix --copt=-Wno-unknown-warning-option --copt=-Wno-stringop-truncation --copt=-Wno-array-parameter
INFO: Found applicable config definition build:posix in file /home/johnny/Projects/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++17 --host_cxxopt=-std=c++17
Loading: 
Loading: 
Loading: 
Loading: 
Loading: 
DEBUG: /home/johnny/.cache/bazel/_bazel_johnny/ec3d842aa30ca8685e26a318bc6b64a6/external/xla/third_party/py/python_repo.bzl:109:10: Using hermetic Python 3.10
Loading: 
Loading: 
Loading: 
Loading: 
DEBUG: /home/johnny/.cache/bazel/_bazel_johnny/ec3d842aa30ca8685e26a318bc6b64a6/external/xla/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'llvm-raw' because it already exists.
Loading: 
Loading: 
Loading: 
Loading: 
Loading: 
Loading: 
Loading: 
DEBUG: /home/johnny/.cache/bazel/_bazel_johnny/ec3d842aa30ca8685e26a318bc6b64a6/external/tsl/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'nvtx_archive' because it already exists.
DEBUG: /home/johnny/.cache/bazel/_bazel_johnny/ec3d842aa30ca8685e26a318bc6b64a6/external/xla/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'jsoncpp_git' because it already exists.
Loading: 
Loading: 
Loading: 
Loading: 
WARNING: The following configs were expanded more than once: [cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
Loading: 
Loading: 2 packages loaded
Analyzing: target //jaxlib/tools:build_wheel (3 packages loaded, 0 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (36 packages loaded, 10 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (55 packages loaded, 270 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (55 packages loaded, 270 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (58 packages loaded, 270 targets configured)
DEBUG: /home/johnny/.cache/bazel/_bazel_johnny/ec3d842aa30ca8685e26a318bc6b64a6/external/tsl/third_party/nccl/hermetic/nccl_redist_init_repository.bzl:73:10: Downloading and extracting https://files.pythonhosted.org/packages/c1/bb/d09dda47c881f9ff504afd6f9ca4f502ded6d8fc2f572cacc5e39da91c28/nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_aarch64.whl
Analyzing: target //jaxlib/tools:build_wheel (114 packages loaded, 5170 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (162 packages loaded, 6299 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (228 packages loaded, 14485 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (240 packages loaded, 17158 targets configured)
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/zlib.net/zlib-1.3.1.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
Analyzing: target //jaxlib/tools:build_wheel (245 packages loaded, 19699 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (245 packages loaded, 19700 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (245 packages loaded, 19700 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (247 packages loaded, 21654 targets configured)
Analyzing: target //jaxlib/tools:build_wheel (247 packages loaded, 21654 targets configured)
INFO: Analyzed target //jaxlib/tools:build_wheel (249 packages loaded, 21752 targets configured).
INFO: Found 1 target...
[0 / 375] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[226 / 1,010] Compiling llvm/lib/Support/Statistic.cpp; 0s local ... (12 actions, 10 running)
[384 / 2,010] Compiling llvm/lib/Support/Statistic.cpp; 1s local ... (11 actions, 8 running)
[418 / 2,784] Compiling llvm/lib/Support/Statistic.cpp; 3s local ... (12 actions, 11 running)
[437 / 2,812] Compiling jaxlib/cpu/lapack_kernels.cc; 3s local ... (12 actions running)
[445 / 2,822] Compiling jaxlib/cpu/lapack_kernels.cc; 4s local ... (11 actions running)
[454 / 2,852] Compiling jaxlib/cpu/lapack_kernels.cc; 5s local ... (12 actions running)
[464 / 2,886] Compiling jaxlib/cpu/lapack_kernels.cc; 6s local ... (12 actions running)
[475 / 2,953] Compiling jaxlib/cpu/lapack_kernels.cc; 7s local ... (12 actions running)
[478 / 2,954] Compiling jaxlib/cpu/lapack_kernels.cc; 8s local ... (12 actions running)
[486 / 2,956] Compiling jaxlib/cpu/lapack_kernels.cc; 9s local ... (12 actions running)
[488 / 2,956] Compiling jaxlib/cpu/lapack_kernels.cc; 10s local ... (12 actions running)
[498 / 2,971] Compiling jaxlib/cpu/lapack_kernels.cc; 11s local ... (12 actions running)
[508 / 3,056] Compiling jaxlib/cpu/lapack_kernels.cc; 12s local ... (12 actions running)
[513 / 3,056] Compiling jaxlib/cpu/lapack_kernels.cc; 13s local ... (12 actions running)
[522 / 3,233] Compiling jaxlib/cpu/lapack_kernels.cc; 14s local ... (12 actions running)
[527 / 3,233] Compiling jaxlib/cpu/lapack_kernels.cc; 15s local ... (12 actions running)
[534 / 3,235] Compiling jaxlib/cpu/lapack_kernels.cc; 16s local ... (12 actions running)
[540 / 3,235] Compiling jaxlib/cpu/lapack_kernels.cc; 17s local ... (12 actions running)
[547 / 3,235] Compiling jaxlib/cpu/lapack_kernels.cc; 18s local ... (12 actions running)
[553 / 3,235] Compiling jaxlib/cpu/lapack_kernels.cc; 19s local ... (12 actions running)
[557 / 3,236] Compiling jaxlib/cpu/lapack_kernels.cc; 20s local ... (12 actions running)
[565 / 3,237] Compiling jaxlib/cpu/lapack_kernels.cc; 22s local ... (12 actions, 11 running)
[573 / 3,241] Compiling jaxlib/cpu/lapack_kernels.cc; 23s local ... (12 actions, 11 running)
[584 / 3,247] Compiling llvm/lib/Support/TimeProfiler.cpp; 3s local ... (12 actions, 11 running)
[588 / 3,247] Compiling llvm/lib/Support/ScopedPrinter.cpp; 3s local ... (12 actions, 11 running)
[594 / 3,302] Compiling llvm/utils/TableGen/IntrinsicEmitter.cpp [for tool]; 4s local ... (12 actions, 11 running)
[598 / 3,334] Compiling llvm/utils/TableGen/IntrinsicEmitter.cpp [for tool]; 5s local ... (12 actions, 11 running)
[607 / 3,349] Compiling llvm/utils/TableGen/IntrinsicEmitter.cpp [for tool]; 6s local ... (12 actions running)
[615 / 3,349] Compiling tsl/platform/cpu_info.cc; 4s local ... (12 actions running)
[621 / 3,349] Compiling src/google/protobuf/compiler/python/generator.cc [for tool]; 4s local ... (11 actions running)
[627 / 3,349] Compiling src/google/protobuf/compiler/python/generator.cc [for tool]; 5s local ... (12 actions running)
[633 / 3,349] Compiling src/compiler/csharp_generator.cc [for tool]; 5s local ... (12 actions running)
[646 / 3,352] Compiling src/google/protobuf/wire_format.cc; 5s local ... (12 actions, 11 running)
[650 / 3,352] Compiling xla/stream_executor/cuda/cuda_diagnostics.cc; 5s local ... (12 actions, 11 running)
[660 / 3,352] Compiling src/compiler/cpp_generator.cc [for tool]; 3s local ... (12 actions, 11 running)
[665 / 3,352] Compiling src/compiler/cpp_generator.cc [for tool]; 4s local ... (12 actions, 11 running)
[671 / 3,352] Compiling src/compiler/cpp_generator.cc [for tool]; 5s local ... (12 actions, 11 running)
[679 / 3,352] Compiling tsl/platform/setround.cc; 3s local ... (12 actions, 11 running)
[691 / 3,356] Compiling llvm/lib/Support/WithColor.cpp [for tool]; 2s local ... (12 actions, 11 running)
[698 / 3,356] Compiling src/google/protobuf/compiler/php/php_generator.cc [for tool]; 3s local ... (12 actions running)
[705 / 3,363] Compiling src/google/protobuf/compiler/php/php_generator.cc [for tool]; 4s local ... (11 actions, 10 running)
[775 / 3,502] Compiling src/google/protobuf/compiler/php/php_generator.cc [for tool]; 5s local ... (12 actions, 11 running)
[781 / 3,502] Compiling src/google/protobuf/compiler/php/php_generator.cc [for tool]; 6s local ... (12 actions, 11 running)
[783 / 3,503] Compiling src/google/protobuf/compiler/php/php_generator.cc [for tool]; 7s local ... (12 actions, 11 running)
[787 / 3,503] Compiling llvm/lib/Support/VirtualFileSystem.cpp [for tool]; 6s local ... (12 actions running)
[791 / 3,503] Compiling llvm/lib/Support/VirtualFileSystem.cpp [for tool]; 7s local ... (12 actions running)
[799 / 3,504] Compiling llvm/lib/Support/VirtualFileSystem.cpp [for tool]; 8s local ... (12 actions running)
[804 / 3,531] Compiling llvm/lib/Support/VirtualFileSystem.cpp [for tool]; 9s local ... (12 actions running)
[809 / 3,531] Compiling xla/permutation_util.cc; 3s local ... (12 actions, 11 running)
[819 / 3,531] Compiling src/google/protobuf/generated_message_tctable_lite.cc [for tool]; 4s local ... (12 actions, 11 running)
[829 / 3,531] Compiling llvm/lib/Support/WithColor.cpp; 2s local ... (12 actions, 11 running)
[833 / 3,531] Compiling xla/array.cc; 2s local ... (12 actions, 11 running)
[838 / 3,531] Compiling xla/array.cc; 3s local ... (12 actions, 11 running)
[849 / 3,548] Compiling llvm/lib/Support/Path.cpp; 4s local ... (12 actions, 11 running)
[856 / 3,548] Compiling xla/tsl/util/command_line_flags.cc; 4s local ... (12 actions, 11 running)
[865 / 3,548] Compiling src/google/protobuf/extension_set.cc [for tool]; 4s local ... (12 actions running)
[872 / 3,557] Compiling src/google/protobuf/compiler/cpp/extension.cc [for tool]; 3s local ... (12 actions running)
[878 / 3,557] Compiling tsl/platform/scanner.cc; 2s local ... (12 actions running)
[882 / 3,557] Compiling tsl/platform/scanner.cc; 3s local ... (12 actions running)
[892 / 3,557] Compiling tsl/platform/stringprintf.cc; 3s local ... (12 actions running)
[898 / 3,558] Compiling src/google/protobuf/compiler/cpp/field.cc [for tool]; 3s local ... (12 actions running)
[903 / 3,558] Compiling src/google/protobuf/util/internal/proto_writer.cc; 2s local ... (12 actions running)
[912 / 3,613] Compiling src/google/protobuf/compiler/objectivec/objectivec_helpers.cc [for tool]; 3s local ... (12 actions, 11 running)
[916 / 3,613] Compiling src/google/protobuf/compiler/objectivec/objectivec_helpers.cc [for tool]; 4s local ... (12 actions, 11 running)
[925 / 3,614] Compiling src/google/protobuf/compiler/cpp/file.cc [for tool]; 4s local ... (12 actions, 11 running)
[927 / 3,614] Compiling src/google/protobuf/compiler/cpp/file.cc [for tool]; 5s local ... (12 actions, 11 running)
[932 / 3,621] Compiling src/google/protobuf/compiler/cpp/file.cc [for tool]; 6s local ... (12 actions running)
[936 / 3,622] Compiling src/google/protobuf/compiler/cpp/file.cc [for tool]; 7s local ... (12 actions running)
[944 / 3,622] Compiling src/google/protobuf/wire_format.cc [for tool]; 4s local ... (12 actions running)
[949 / 3,622] Compiling absl/strings/cord.cc; 4s local ... (12 actions running)
[960 / 3,627] Compiling src/google/protobuf/compiler/objectivec/objectivec_file.cc [for tool]; 3s local ... (12 actions, 11 running)
[967 / 3,628] Compiling src/google/protobuf/compiler/objectivec/objectivec_file.cc [for tool]; 4s local ... (12 actions, 11 running)
[976 / 3,629] Compiling src/google/protobuf/extension_set.cc; 3s local ... (12 actions, 11 running)
[983 / 3,641] Compiling src/google/protobuf/extension_set.cc; 4s local ... (12 actions, 11 running)
[987 / 3,641] Compiling src/google/protobuf/extension_set.cc; 5s local ... (11 actions, 10 running)
[993 / 3,641] Compiling src/google/protobuf/compiler/cpp/helpers.cc [for tool]; 4s local ... (12 actions, 11 running)
[999 / 3,641] Compiling src/google/protobuf/compiler/cpp/helpers.cc [for tool]; 5s local ... (12 actions, 11 running)
[1,005 / 3,641] Compiling src/google/protobuf/util/message_differencer.cc [for tool]; 3s local ... (12 actions, 11 running)
[1,011 / 3,642] Compiling src/google/protobuf/util/message_differencer.cc [for tool]; 4s local ... (12 actions running)
[1,018 / 3,644] Compiling src/google/protobuf/util/message_differencer.cc [for tool]; 5s local ... (12 actions running)
[1,119 / 5,195] Compiling src/google/protobuf/util/message_differencer.cc [for tool]; 6s local ... (12 actions, 11 running)
[1,707 / 5,201] Compiling src/google/protobuf/util/field_mask_util.cc; 2s local ... (12 actions, 11 running)
[2,334 / 5,202] Compiling src/google/protobuf/compiler/objectivec/objectivec_enum.cc [for tool]; 1s local ... (12 actions, 11 running)
[2,688 / 5,294] Compiling src/google/protobuf/compiler/objectivec/objectivec_enum.cc [for tool]; 2s local ... (12 actions, 11 running)
[2,696 / 5,300] Compiling tsl/platform/random.cc; 4s local ... (12 actions, 11 running)
[2,710 / 5,671] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 5s local ... (12 actions, 11 running)
[2,721 / 5,671] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 6s local ... (12 actions, 11 running)
[2,732 / 5,671] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 7s local ... (12 actions, 11 running)
[2,742 / 5,671] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 8s local ... (12 actions, 11 running)
[2,755 / 5,672] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 9s local ... (12 actions, 11 running)
[2,768 / 5,673] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 10s local ... (12 actions, 11 running)
[2,775 / 5,673] Compiling src/google/protobuf/compiler/cpp/message.cc [for tool]; 11s local ... (12 actions, 11 running)
[2,786 / 5,677] Compiling xla/parse_flags_from_env.cc; 4s local ... (12 actions, 11 running)
[2,799 / 5,679] Compiling xla/parse_flags_from_env.cc; 5s local ... (12 actions, 11 running)
[2,814 / 5,679] Compiling src/google/protobuf/compiler/cpp/message_field.cc [for tool]; 3s local ... (12 actions, 11 running)
[2,825 / 5,686] Compiling src/google/protobuf/compiler/cpp/message_field.cc [for tool]; 4s local ... (12 actions, 11 running)
[2,840 / 5,686] Compiling src/google/protobuf/util/internal/protostream_objectsource.cc [for tool]; 3s local ... (12 actions, 11 running)
[2,854 / 5,687] Compiling src/google/protobuf/text_format.cc; 3s local ... (12 actions, 11 running)
[2,867 / 5,701] Compiling src/google/protobuf/text_format.cc; 4s local ... (12 actions, 11 running)
[2,877 / 5,704] Compiling src/google/protobuf/text_format.cc; 5s local ... (12 actions, 11 running)
[2,892 / 5,704] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 3s local ... (12 actions running)
[2,902 / 5,705] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 4s local ... (12 actions, 11 running)
[2,917 / 5,710] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 5s local ... (12 actions, 11 running)
[2,928 / 5,710] Compiling llvm/lib/Demangle/ItaniumDemangle.cpp; 6s local ... (12 actions, 11 running)
[2,939 / 5,710] Compiling src/google/protobuf/compiler/cpp/parse_function_generator.cc [for tool]; 5s local ... (12 actions, 11 running)
[2,949 / 5,710] Compiling src/google/protobuf/compiler/cpp/parse_function_generator.cc [for tool]; 6s local ... (12 actions, 11 running)
[2,969 / 5,713] Compiling llvm/lib/Demangle/MicrosoftDemangle.cpp; 2s local ... (12 actions, 11 running)
[2,980 / 6,318] Compiling src/google/protobuf/source_context.pb.cc; 2s local ... (12 actions, 11 running)
[2,991 / 7,427] Compiling llvm/lib/Support/ScopedPrinter.cpp [for tool]; 2s local ... (12 actions, 11 running)
[3,003 / 7,541] Compiling src/google/protobuf/compiler/cpp/primitive_field.cc [for tool]; 3s local ... (12 actions, 11 running)
[3,011 / 7,541] Compiling llvm/lib/DebugInfo/MSF/MappedBlockStream.cpp; 2s local ... (12 actions, 11 running)
[3,025 / 7,546] Compiling llvm/lib/DebugInfo/MSF/MappedBlockStream.cpp; 3s local ... (12 actions, 11 running)
[3,031 / 7,546] Compiling llvm/lib/Remarks/YAMLRemarkSerializer.cpp; 3s local ... (12 actions, 11 running)
[3,040 / 7,546] Compiling src/google/protobuf/util/internal/default_value_objectwriter.cc [for tool]; 2s local ... (12 actions running)
[3,045 / 7,546] Compiling src/google/protobuf/util/internal/default_value_objectwriter.cc [for tool]; 3s local ... (12 actions running)
[3,054 / 7,546] Compiling llvm/lib/Remarks/YAMLRemarkParser.cpp; 3s local ... (12 actions, 11 running)
[3,063 / 7,547] Compiling src/google/protobuf/compiler/cpp/string_field.cc [for tool]; 3s local ... (12 actions, 11 running)
[3,071 / 7,547] Compiling src/google/protobuf/util/internal/datapiece.cc [for tool]; 2s local ... (12 actions, 11 running)
[3,078 / 7,547] Compiling src/google/protobuf/util/internal/datapiece.cc [for tool]; 3s local ... (12 actions, 11 running)
[3,091 / 7,717] Compiling llvm/lib/Bitstream/Reader/BitstreamReader.cpp; 3s local ... (12 actions, 11 running)
[3,098 / 7,717] Compiling src/google/protobuf/compiler/importer.cc; 2s local ... (12 actions, 11 running)
[3,106 / 7,717] Compiling src/google/protobuf/compiler/java/message_builder.cc [for tool]; 2s local ... (12 actions, 11 running)
[3,113 / 7,717] Compiling src/google/protobuf/compiler/java/message_builder.cc [for tool]; 3s local ... (12 actions, 11 running)
[3,122 / 7,717] Compiling src/google/protobuf/compiler/parser.cc; 3s local ... (12 actions, 11 running)
[3,132 / 7,717] Compiling src/google/protobuf/compiler/parser.cc; 4s local ... (12 actions, 11 running)
[3,140 / 7,717] Compiling src/google/protobuf/compiler/parser.cc; 5s local ... (12 actions, 11 running)
[3,147 / 7,717] Compiling mlir/lib/Support/Timing.cpp; 4s local ... (12 actions, 11 running)
[3,153 / 7,717] Compiling src/google/protobuf/compiler/csharp/csharp_field_base.cc [for tool]; 3s local ... (12 actions, 11 running)
[3,161 / 7,717] Compiling llvm/lib/Support/APFloat.cpp; 3s local ... (12 actions, 11 running)
[3,165 / 7,717] Compiling llvm/lib/Support/APFloat.cpp; 4s local ... (12 actions running)
[3,173 / 7,717] Compiling llvm/lib/Support/APFloat.cpp; 5s local ... (12 actions running)
[3,180 / 7,717] Compiling llvm/lib/Support/APFloat.cpp; 6s local ... (12 actions running)
[3,190 / 7,717] Compiling src/google/protobuf/descriptor.cc; 6s local ... (12 actions running)
[3,205 / 7,717] Compiling src/google/protobuf/descriptor.cc; 7s local ... (12 actions, 11 running)
[3,212 / 7,717] Compiling src/google/protobuf/descriptor.cc; 8s local ... (12 actions, 11 running)
[3,220 / 7,717] Compiling src/google/protobuf/descriptor.cc; 9s local ... (12 actions, 11 running)
[3,228 / 7,717] Compiling src/google/protobuf/descriptor.cc; 10s local ... (12 actions, 11 running)
[3,234 / 7,717] Compiling src/google/protobuf/descriptor.cc; 11s local ... (12 actions, 11 running)
[3,239 / 7,717] Compiling src/google/protobuf/descriptor.cc; 12s local ... (12 actions, 11 running)
[3,247 / 7,717] Compiling src/google/protobuf/descriptor.cc; 13s local ... (12 actions, 11 running)
[3,265 / 7,720] Compiling src/google/protobuf/descriptor.cc; 15s local ... (12 actions, 11 running)
[3,269 / 7,720] Compiling src/google/protobuf/descriptor.cc; 16s local ... (12 actions, 11 running)
[3,273 / 7,720] Compiling src/google/protobuf/descriptor.cc; 17s local ... (12 actions running)
[3,278 / 7,720] Compiling src/google/protobuf/descriptor.cc; 18s local ... (12 actions, 11 running)
[3,284 / 7,720] Compiling src/google/protobuf/descriptor.cc; 19s local ... (12 actions, 11 running)
[3,291 / 7,720] Compiling src/crypto/fipsmodule/bcm.c; 5s local ... (12 actions, 11 running)
[3,298 / 7,720] Compiling src/crypto/fipsmodule/bcm.c; 7s local ... (12 actions, 11 running)
[3,304 / 7,720] Compiling src/crypto/fipsmodule/bcm.c; 8s local ... (12 actions running)
[3,310 / 7,749] Compiling src/crypto/fipsmodule/bcm.c; 9s local ... (12 actions running)
[3,313 / 7,749] Compiling src/crypto/fipsmodule/bcm.c; 10s local ... (12 actions, 11 running)
[3,318 / 7,762] Compiling src/crypto/fipsmodule/bcm.c; 12s local ... (12 actions, 11 running)
[3,322 / 7,762] Compiling src/crypto/fipsmodule/bcm.c; 13s local ... (12 actions, 11 running)
[3,325 / 7,762] Compiling src/crypto/fipsmodule/bcm.c; 14s local ... (12 actions, 11 running)
[3,330 / 7,762] Compiling src/crypto/fipsmodule/bcm.c; 15s local ... (12 actions running)
[3,334 / 7,762] Compiling llvm/lib/Support/CommandLine.cpp; 8s local ... (12 actions running)
[3,345 / 7,775] Compiling src/google/protobuf/compiler/parser.cc [for tool]; 6s local ... (12 actions, 11 running)
[3,355 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 6s local ... (12 actions, 11 running)
[3,363 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 7s local ... (12 actions, 11 running)
[3,372 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 8s local ... (12 actions, 11 running)
[3,377 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 9s local ... (12 actions, 11 running)
[3,387 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 10s local ... (12 actions, 11 running)
[3,392 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 11s local ... (12 actions, 11 running)
[3,396 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 12s local ... (12 actions running)
[3,401 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 13s local ... (12 actions running)
[3,413 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 14s local ... (12 actions running)
[3,420 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 15s local ... (12 actions running)
[3,428 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 17s local ... (12 actions, 11 running)
[3,440 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 18s local ... (12 actions, 11 running)
[3,447 / 7,775] Compiling src/google/protobuf/descriptor.cc [for tool]; 19s local ... (12 actions, 11 running)
[3,458 / 7,775] Compiling llvm/lib/Support/APInt.cpp [for tool]; 4s local ... (12 actions, 11 running)
[3,475 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 3s local ... (12 actions, 11 running)
[3,523 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 4s local ... (12 actions, 11 running)
[3,546 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 5s local ... (12 actions, 11 running)
[3,552 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 6s local ... (12 actions, 11 running)
[3,559 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 7s local ... (12 actions, 11 running)
[3,573 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 8s local ... (12 actions, 11 running)
[3,586 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 9s local ... (12 actions, 11 running)
[3,598 / 7,775] Compiling llvm/lib/TableGen/Record.cpp [for tool]; 10s local ... (12 actions, 11 running)
[3,609 / 7,812] Compiling llvm/lib/Support/CommandLine.cpp [for tool]; 6s local ... (12 actions, 11 running)
[3,614 / 7,812] Compiling llvm/lib/Support/CommandLine.cpp [for tool]; 7s local ... (12 actions, 11 running)
[3,619 / 7,812] Compiling llvm/lib/Support/CommandLine.cpp [for tool]; 8s local ... (12 actions running)
[3,622 / 7,812] Compiling xla/window_util.cc; 5s local ... (12 actions running)
[3,634 / 7,812] Compiling src/ssl/tls13_server.cc; 1s local ... (12 actions running)
[3,639 / 7,817] Compiling llvm/lib/Support/GlobPattern.cpp [for tool]; 2s local ... (12 actions running)
[3,645 / 7,834] Compiling external/tsl/tsl/protobuf/dnn.pb.cc; 2s local ... (12 actions, 11 running)
[3,647 / 7,834] Compiling external/xla/xla/xla_data.pb.cc; 4s local ... (12 actions, 11 running)
[3,656 / 7,844] Compiling external/xla/xla/xla_data.pb.cc; 5s local ... (12 actions, 11 running)
[3,663 / 7,844] Compiling external/xla/xla/xla_data.pb.cc; 6s local ... (12 actions, 11 running)
[3,668 / 7,844] Compiling external/xla/xla/xla_data.pb.cc; 7s local ... (12 actions, 11 running)
[3,672 / 7,844] Compiling external/xla/xla/xla_data.pb.cc; 8s local ... (12 actions running)
[3,677 / 7,844] Compiling external/xla/xla/xla_data.pb.cc; 9s local ... (12 actions running)
[3,684 / 7,844] Compiling external/xla/xla/xla_data.pb.cc; 10s local ... (12 actions, 11 running)
[3,686 / 7,844] Compiling xla/debug_options_flags.cc; 12s local ... (12 actions, 11 running)
[3,695 / 7,847] Compiling xla/debug_options_flags.cc; 13s local ... (12 actions, 11 running)
[3,701 / 7,847] Compiling xla/debug_options_flags.cc; 14s local ... (12 actions, 11 running)
[3,704 / 7,847] Compiling xla/debug_options_flags.cc; 15s local ... (12 actions running)
[3,708 / 7,847] Compiling xla/debug_options_flags.cc; 16s local ... (12 actions running)
[3,715 / 7,848] Compiling xla/debug_options_flags.cc; 18s local ... (12 actions, 11 running)
[3,716 / 7,848] Compiling xla/debug_options_flags.cc; 19s local ... (12 actions, 11 running)
[3,724 / 7,849] Compiling tsl/platform/default/posix_file_system.cc; 6s local ... (12 actions, 11 running)
[3,730 / 7,849] Compiling xla/service/compilation_environments.cc; 7s local ... (12 actions running)
[3,732 / 7,849] Compiling xla/service/compilation_environments.cc; 8s local ... (12 actions running)
[3,736 / 7,849] Compiling xla/service/compilation_environments.cc; 9s local ... (12 actions running)
[3,742 / 7,849] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 8s local ... (12 actions, 11 running)
[3,747 / 7,851] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 9s local ... (12 actions, 11 running)
[3,753 / 7,851] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 10s local ... (12 actions, 11 running)
[3,757 / 7,851] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 12s local ... (12 actions, 11 running)
[3,760 / 7,851] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 13s local ... (12 actions, 11 running)
[3,764 / 7,851] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 14s local ... (12 actions running)
[3,767 / 7,851] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 15s local ... (12 actions running)
[3,773 / 7,852] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 17s local ... (12 actions, 11 running)
[3,776 / 7,853] Compiling llvm/utils/TableGen/AsmMatcherEmitter.cpp [for tool]; 18s local ... (12 actions running)
[3,781 / 7,853] Compiling xla/python/profiler/internal/python_hooks.cc; 7s local ... (12 actions, 11 running)
[3,784 / 7,853] Compiling xla/python/profiler/internal/python_hooks.cc; 8s local ... (12 actions, 11 running)
[3,785 / 7,853] Compiling xla/python/profiler/internal/python_hooks.cc; 10s local ... (12 actions running)
ERROR: /home/johnny/Projects/jax/jaxlib/cuda/BUILD:75:13: Compiling jaxlib/gpu/make_batch_pointers.cu.cc failed: undeclared inclusion(s) in rule '//jaxlib/cuda:cuda_make_batch_pointers':
this rule is missing dependency declarations for the following files included by 'jaxlib/gpu/make_batch_pointers.cu.cc':
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_runtime.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/host_config.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/builtin_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/device_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/host_defines.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/driver_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/vector_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/surface_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/texture_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/library_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/channel_descriptor.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_runtime_api.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_device_runtime_api.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/driver_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/vector_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/vector_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/common_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/math_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/math_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/device_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/device_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/device_atomic_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/device_atomic_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/device_double_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/device_double_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_20_atomic_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_20_atomic_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_32_atomic_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_32_atomic_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_35_atomic_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_60_atomic_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_60_atomic_functions.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_20_intrinsics.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_20_intrinsics.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_30_intrinsics.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_30_intrinsics.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_32_intrinsics.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_32_intrinsics.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_35_intrinsics.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_61_intrinsics.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/sm_61_intrinsics.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/sm_70_rt.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/sm_70_rt.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/sm_80_rt.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/sm_80_rt.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/sm_90_rt.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/sm_90_rt.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/texture_indirect_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/surface_indirect_functions.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/crt/cudacc_ext.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/device_launch_parameters.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_result.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_version.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_stdint.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_activity.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_callbacks.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_events.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_metrics.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_driver_cbid.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_runtime_cbid.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cupti_nvtx_cbid.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/generated_cuda_runtime_api_meta.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/generated_cuda_meta.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/type_traits'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/__config'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__config'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_fp16.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_fp16.hpp'
  'external/cuda_nvcc/targets/aarch64-linux/include/nv/target'
  'external/cuda_nvcc/targets/aarch64-linux/include/nv/detail/__target_macros'
  'external/cuda_nvcc/targets/aarch64-linux/include/nv/detail/__preprocessor'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/__pragma_push'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__pragma_push'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__undef_macros'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/type_traits'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__assert'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__verbose_abort'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__availability'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__functional/identity.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/forward.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_reference.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/integral_constant.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_reference.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/cstddef'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__cuda/cstddef_prelude.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/enable_if.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_integral.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_cv.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_const.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_volatile.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/version'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__functional/invoke.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/add_lvalue_reference.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_referenceable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_same.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/apply_cv.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_const.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_volatile.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/conditional.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/decay.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/add_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_void.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_cvref.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_array.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_function.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_extent.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_base_of.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_core_convertible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_member_function_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_member_object_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_reference_wrapper.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/nat.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/declval.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__memory/addressof.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/add_const.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/add_cv.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/add_rvalue_reference.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/add_volatile.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/aligned_storage.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/type_list.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/aligned_union.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/alignment_of.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/can_extract_key.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/pair.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_const_ref.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/common_reference.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/common_type.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/void_t.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/copy_cv.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/copy_cvref.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/disjunction.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_convertible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_scalar.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_arithmetic.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_floating_point.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_enum.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_class.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_union.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_member_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_null_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/conjunction.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/dependent_type.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/extent.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/has_unique_object_representation.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_all_extents.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/has_virtual_destructor.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_abstract.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_aggregate.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_allocator.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_bounded_array.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_callable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_char_like_type.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_standard_layout.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivial.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_copyable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_default_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_compound.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_fundamental.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_constant_evaluated.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_destructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/negation.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_copy_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_copy_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_default_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_empty.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_final.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_implicitly_default_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_literal_type.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_move_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_move_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_convertible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/lazy.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_copy_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_copy_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_default_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_destructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_move_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_nothrow_move_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_object.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_pod.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_copy_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_copy_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_destructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_polymorphic.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_primary_template.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_valid_expansion.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_scoped_enum.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/underlying_type.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_signed_integer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_signed.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_swappable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_move_assignable.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_trivially_move_constructible.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_unbounded_array.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_unsigned_integer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/is_unsigned.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/make_const_lvalue_ref.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/make_32_64_or_128_bit.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/make_unsigned.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/cstdint'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__cuda/cstdint_prelude.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/climits'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__cuda/climits_prelude.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/make_signed.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/maybe_const.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/promote.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/rank.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/remove_pointer.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/result_of.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__type_traits/type_identity.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/convert_to_integral.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/move.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/swap.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/__pragma_pop'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__pragma_pop'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/atomic'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/atomic'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/atomic'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__debug'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__threading_support'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__cuda/atomic_prelude.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/cassert'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__functional/hash.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__functional/unary_function.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/hash.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/pair.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__functional/unwrap_ref.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/get.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/array.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/tuple.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/tuple_element.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/tuple_indices.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/integer_sequence.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/tuple_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/sfinae_helpers.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/make_tuple_types.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/apply_cv.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/tuple_size.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/tuple_like.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__tuple_dir/structured_bindings.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__utility/piecewise_construct.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/chrono'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/ctime'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/limits'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/ratio'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__cuda/chrono.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/iosfwd'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/string.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__fwd/memory_resource.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/atomic_scopes.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/atomic_cuda.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/atomic_gcc.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/atomic_base.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/cxx_atomic.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/atomic_cuda_generated.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/support/atomic/atomic_cuda_derived.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda/std/detail/libcxx/include/__cuda/atomic.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cooperative_groups/details/partitioning.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cooperative_groups/details/invoke.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_bf16.h'
  'external/cuda_nvcc/targets/aarch64-linux/include/cuda_bf16.hpp'
Target //jaxlib/tools:build_wheel failed to build
INFO: Elapsed time: 305.538s, Critical Path: 41.20s
INFO: 3800 processes: 2079 internal, 1721 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
INFO:__main__:subprocess raised: Command '['./bazel-6.5.0-linux-arm64', 'run', '--verbose_failures=true', '--repo_env=LOCAL_CUDA_PATH=/usr/local/cuda-12.2', '--repo_env=LOCAL_CUDNN_PATH=/opt/nvidia/cudnn/', '//jaxlib/tools:build_wheel', '--', '--output_path=/home/johnny/Projects/jax/dist', '--jaxlib_git_hash=1594d2f30fdbfebf693aba4a2b264e4a3e52acc6', '--cpu=aarch64']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/home/johnny/Projects/jax/build/build.py", line 721, in <module>
    main()
  File "/home/johnny/Projects/jax/build/build.py", line 672, in main
    shell(build_cpu_wheel_command)
  File "/home/johnny/Projects/jax/build/build.py", line 45, in shell
    output = subprocess.check_output(cmd)
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-6.5.0-linux-arm64', 'run', '--verbose_failures=true', '--repo_env=LOCAL_CUDA_PATH=/usr/local/cuda-12.2', '--repo_env=LOCAL_CUDNN_PATH=/opt/nvidia/cudnn/', '//jaxlib/tools:build_wheel', '--', '--output_path=/home/johnny/Projects/jax/dist', '--jaxlib_git_hash=1594d2f30fdbfebf693aba4a2b264e4a3e52acc6', '--cpu=aarch64']' returned non-zero exit status 1.
(py310) johnny@johnny-jetson:~/Projects/jax$
dusty-nv commented 5 days ago

@johnnynunez that was because /usr/local/cuda/include and /usr/local/cuda/lib64 are symlinks under /usr/local/cuda/targets, and needed replaced with the hard files, as it does not seem bazel could follow the links.

https://github.com/dusty-nv/jetson-containers/commit/2f22c19b04b3e4468430373887b52c1218225246#diff-9e8906da5112ddbb028424f777cf002d780fe985c9bde466d8b70c092e010250R14

ybaturina commented 5 days ago

hi @johnnynunez, do CUDA local path /usr/local/cuda-12.2 and CUDNN local path /opt/nvidia/cudnn have required folder structure as described in the docs?

9L4QCSjg8XtydRW

dusty-nv commented 5 days ago

@ybaturina on Jetson the packages come slightly differently, we moved things around for the build to succeed 👍

It is not clear from the build though when the local CUDA/cuDNN is properly detected until an error is hit.

johnnynunez commented 5 days ago

@ybaturina @dfm @justinjfu Yeah the problem is that XLA consider SBSA as unique aarch64, and SBSA is ARM64 CPU servers. aarch64 is considered arm64 but for tegra chips. I've tried to facilitate the build from jax in jetson devices. https://github.com/openxla/xla/pull/16905

summary: SBSA is for aarch64 but for arm servers. aarch64 for nvidia is formally tegra cpus, like jetson.

johnnynunez commented 5 days ago

hi @johnnynunez, do CUDA local path /usr/local/cuda-12.2 and CUDNN local path /opt/nvidia/cudnn have required folder structure as described in the docs?

9L4QCSjg8XtydRW

yes it is done. I've copied files to /opt/nvidia/cudnn, but for cuda I did symbolic link because the real path on cuda installation is lib64

ybaturina commented 5 days ago

hi @dusty-nv, will it help if CUDA repo rules throw an error if required folders are not available in CUDA/CUDNN/NCCL paths?

@johnnynunez Ill prepare the instructions how to download NVIDIA redistributions from NVIDIA site.

ybaturina commented 5 days ago

@johnnynunez I think you can use custom CUDA/CUDNN configuration:

E.g. you can define a dict in the WORKSPACE file like this:

   _CUSTOM_CUDA_REDISTRIBUTIONS = {
      "cuda_cccl": {
         "linux-sbsa": {
            "relative_path": "cuda_cccl/linux-aarch64/cuda_cccl-linux-aarch64-12.4.127-archive.tar.xz"
         }
      },
   }

The keys in the dict are linux-sbsa, but the relative paths have folder linux-aarch64. This way you'll be able to download correct redistribution versions for Jetson platform.

Then at this line you can use this:

cuda_redistributions = CUDA_REDISTRIBUTIONS | _CUSTOM_CUDA_REDISTRIBUTIONS.

dusty-nv commented 5 days ago

hi @dusty-nv, will it help if CUDA repo rules throw an error if required folders are not available in CUDA/CUDNN/NCCL paths?

Yes, that would be useful, thanks!

ybaturina commented 5 days ago

Yes, that would be useful, thanks!

I've created https://github.com/openxla/xla/pull/17107, going to merge it soon.

johnnynunez commented 4 days ago

Solved, llvm 18 is required