search

NVIDIA / thrust

[ARCHIVED] The C++ parallel algorithms library. See https://github.com/NVIDIA/cccl
Other
4.92k stars 758 forks source link

nvcc fatal : Unsupported gpu architecture 'compute_90' for $ cmake --build . -j ${NUM_JOBS} #1948

Closed monajalal closed 1 year ago

monajalal commented 1 year ago

Could you please help with fixing this issue?

(oneposeplus) mona@ard-gpu-01:~/thrust/build$ cmake --build . -j ${NUM_JOBS}
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.bucket_sort2d.dir/bucket_sort2d.cu.o
[  0%] Building CUDA object testing/unittest/CMakeFiles/thrust.test.framework.dir/testframework.cu.o
[  0%] Building CUDA object testing/unittest/CMakeFiles/thrust.test.framework.dir/cuda/testframework.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.arbitrary_transformation.dir/arbitrary_transformation.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.bounding_box.dir/bounding_box.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.basic_vector.dir/basic_vector.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.constant_iterator.dir/constant_iterator.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.device_ptr.dir/device_ptr.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.counting_iterator.dir/counting_iterator.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.expand.dir/expand.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.dot_products_with_zip.dir/dot_products_with_zip.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.bucket_sort2d.dir/build.make:76: examples/CMakeFiles/thrust.example.bucket_sort2d.dir/bucket_sort2d.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.basic_vector.dir/build.make:76: examples/CMakeFiles/thrust.example.basic_vector.dir/basic_vector.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [testing/unittest/CMakeFiles/thrust.test.framework.dir/build.make:76: testing/unittest/CMakeFiles/thrust.test.framework.dir/testframework.cu.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.constant_iterator.dir/build.make:76: examples/CMakeFiles/thrust.example.constant_iterator.dir/constant_iterator.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.expand.dir/build.make:76: examples/CMakeFiles/thrust.example.expand.dir/expand.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.device_ptr.dir/build.make:76: examples/CMakeFiles/thrust.example.device_ptr.dir/device_ptr.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.bounding_box.dir/build.make:76: examples/CMakeFiles/thrust.example.bounding_box.dir/bounding_box.cu.o] Error 1
gmake[2]: *** [examples/CMakeFiles/thrust.example.dot_products_with_zip.dir/build.make:76: examples/CMakeFiles/thrust.example.dot_products_with_zip.dir/dot_products_with_zip.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [testing/unittest/CMakeFiles/thrust.test.framework.dir/build.make:90: testing/unittest/CMakeFiles/thrust.test.framework.dir/cuda/testframework.cu.o] Error 1
gmake[2]: *** [examples/CMakeFiles/thrust.example.arbitrary_transformation.dir/build.make:76: examples/CMakeFiles/thrust.example.arbitrary_transformation.dir/arbitrary_transformation.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:10803: testing/unittest/CMakeFiles/thrust.test.framework.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:15034: examples/CMakeFiles/thrust.example.arbitrary_transformation.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15086: examples/CMakeFiles/thrust.example.basic_vector.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15138: examples/CMakeFiles/thrust.example.bounding_box.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15190: examples/CMakeFiles/thrust.example.bucket_sort2d.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15242: examples/CMakeFiles/thrust.example.constant_iterator.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15346: examples/CMakeFiles/thrust.example.device_ptr.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15450: examples/CMakeFiles/thrust.example.dot_products_with_zip.dir/all] Error 2
gmake[1]: *** [CMakeFiles/Makefile2:15502: examples/CMakeFiles/thrust.example.expand.dir/all] Error 2
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.lexicographical_sort.dir/lexicographical_sort.cu.o
[  0%] Building CUDA object examples/CMakeFiles/thrust.example.fill_copy_sequence.dir/fill_copy_sequence.cu.o
[  1%] Building CUDA object examples/CMakeFiles/thrust.example.histogram.dir/histogram.cu.o
[  1%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/allocate_unique.h.cu.o
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/adjacent_difference.h.cu.o
[  1%] Building CUDA object examples/CMakeFiles/thrust.example.discrete_voronoi.dir/discrete_voronoi.cu.o
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/async/copy.h.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_90'
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/addressof.h.cu.o
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/advance.h.cu.o
gmake[2]: *** [examples/CMakeFiles/thrust.example.lexicographical_sort.dir/build.make:76: examples/CMakeFiles/thrust.example.lexicographical_sort.dir/lexicographical_sort.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15710: examples/CMakeFiles/thrust.example.lexicographical_sort.dir/all] Error 2
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.fill_copy_sequence.dir/build.make:76: examples/CMakeFiles/thrust.example.fill_copy_sequence.dir/fill_copy_sequence.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15554: examples/CMakeFiles/thrust.example.fill_copy_sequence.dir/all] Error 2
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/async/for_each.h.cu.o
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/async/reduce.h.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.discrete_voronoi.dir/build.make:76: examples/CMakeFiles/thrust.example.discrete_voronoi.dir/discrete_voronoi.cu.o] Error 1
[  2%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/async/scan.h.cu.o
gmake[1]: *** [CMakeFiles/Makefile2:15398: examples/CMakeFiles/thrust.example.discrete_voronoi.dir/all] Error 2
gmake[2]: *** [examples/CMakeFiles/thrust.example.counting_iterator.dir/build.make:76: examples/CMakeFiles/thrust.example.counting_iterator.dir/counting_iterator.cu.o] Error 1
gmake[2]: *** [examples/CMakeFiles/thrust.example.histogram.dir/build.make:76: examples/CMakeFiles/thrust.example.histogram.dir/histogram.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15294: examples/CMakeFiles/thrust.example.counting_iterator.dir/all] Error 2
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:118: CMakeFiles/thrust.headers.dir/headers/thrust/allocate_unique.h.cu.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:90: CMakeFiles/thrust.headers.dir/headers/thrust/adjacent_difference.h.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15606: examples/CMakeFiles/thrust.example.histogram.dir/all] Error 2
nvcc fatal   : Unsupported gpu architecture 'compute_90'
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:76: CMakeFiles/thrust.headers.dir/headers/thrust/addressof.h.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:104: CMakeFiles/thrust.headers.dir/headers/thrust/advance.h.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:146: CMakeFiles/thrust.headers.dir/headers/thrust/async/for_each.h.cu.o] Error 1
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:132: CMakeFiles/thrust.headers.dir/headers/thrust/async/copy.h.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:160: CMakeFiles/thrust.headers.dir/headers/thrust/async/reduce.h.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:174: CMakeFiles/thrust.headers.dir/headers/thrust/async/scan.h.cu.o] Error 1
[  2%] Building CUDA object examples/CMakeFiles/thrust.example.max_abs_diff.dir/max_abs_diff.cu.o
[  2%] Building CUDA object examples/CMakeFiles/thrust.example.lambda.dir/lambda.cu.o
[  3%] Building CUDA object examples/CMakeFiles/thrust.example.minimal_custom_backend.dir/minimal_custom_backend.cu.o
[  3%] Building CUDA object examples/CMakeFiles/thrust.example.monte_carlo.dir/monte_carlo.cu.o
[  3%] Building CUDA object examples/CMakeFiles/thrust.example.minmax.dir/minmax.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_90'
[  3%] Building CUDA object CMakeFiles/thrust.headers.dir/headers/thrust/async/sort.h.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.max_abs_diff.dir/build.make:76: examples/CMakeFiles/thrust.example.max_abs_diff.dir/max_abs_diff.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15762: examples/CMakeFiles/thrust.example.max_abs_diff.dir/all] Error 2
gmake[2]: *** [examples/CMakeFiles/thrust.example.lambda.dir/build.make:76: examples/CMakeFiles/thrust.example.lambda.dir/lambda.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15658: examples/CMakeFiles/thrust.example.lambda.dir/all] Error 2
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.minimal_custom_backend.dir/build.make:76: examples/CMakeFiles/thrust.example.minimal_custom_backend.dir/minimal_custom_backend.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15814: examples/CMakeFiles/thrust.example.minimal_custom_backend.dir/all] Error 2
nvcc fatal   : Unsupported gpu architecture 'compute_90'
[  3%] Building CUDA object examples/CMakeFiles/thrust.example.mode.dir/mode.cu.o
gmake[2]: *** [examples/CMakeFiles/thrust.example.monte_carlo.dir/build.make:76: examples/CMakeFiles/thrust.example.monte_carlo.dir/monte_carlo.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15970: examples/CMakeFiles/thrust.example.monte_carlo.dir/all] Error 2
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [CMakeFiles/thrust.headers.dir/build.make:188: CMakeFiles/thrust.headers.dir/headers/thrust/async/sort.h.cu.o] Error 1
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[1]: *** [CMakeFiles/Makefile2:1255: CMakeFiles/thrust.headers.dir/all] Error 2
gmake[2]: *** [examples/CMakeFiles/thrust.example.minmax.dir/build.make:76: examples/CMakeFiles/thrust.example.minmax.dir/minmax.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15866: examples/CMakeFiles/thrust.example.minmax.dir/all] Error 2
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.mode.dir/build.make:76: examples/CMakeFiles/thrust.example.mode.dir/mode.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:15918: examples/CMakeFiles/thrust.example.mode.dir/all] Error 2
[  3%] Building CUDA object examples/CMakeFiles/thrust.example.monte_carlo_disjoint_sequences.dir/monte_carlo_disjoint_sequences.cu.o
nvcc fatal   : Unsupported gpu architecture 'compute_90'
gmake[2]: *** [examples/CMakeFiles/thrust.example.monte_carlo_disjoint_sequences.dir/build.make:76: examples/CMakeFiles/thrust.example.monte_carlo_disjoint_sequences.dir/monte_carlo_disjoint_sequences.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:16022: examples/CMakeFiles/thrust.example.monte_carlo_disjoint_sequences.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2

system info

(oneposeplus) mona@ard-gpu-01:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

(oneposeplus) mona@ard-gpu-01:~$ uname -a
Linux ard-gpu-01 5.19.0-42-generic #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Apr 21 16:51:08 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
(oneposeplus) mona@ard-gpu-01:~$ lsb_release -a
LSB Version:    core-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:    22.04
Codename:   jammy
(oneposeplus) mona@ard-gpu-01:~$ gcc --version
gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(oneposeplus) mona@ard-gpu-01:~$ g++ --version
g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

(oneposeplus) mona@ard-gpu-01:~$ cmake --version
cmake version 3.22.1

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Screenshot from 2023-05-30 10-25-36

Screenshot from 2023-05-30 10-30-35 Screenshot from 2023-05-30 10-30-47

miscco commented 1 year ago

Hi @monajalal,

It seems that this is not an issue from the thrust side, but nvcc being too old to support compute_90. You can see that nvcc 11.7 does not support the sm_90 flag here https://godbolt.org/z/j5Ge8rrba

I would recommend to upgrade your cuda toolkit to the latest version

monajalal commented 1 year ago

I need to use CUDA 11.7 for PyTorch 1.13.0

jrhemstad commented 1 year ago

compute_90 requires CUDA 11.8 or newer.

If you want to use a Hopper GPU with 11.7, you can compile for an older architecture (like compute_80) and then rely on PTX JIT to JIT compile for running on Hopper.

If your library is large, then relying on PTX JIT can take quite a while on the first run because it has to JIT compile your entire library's PTX code.

monajalal commented 1 year ago

I ended up installing ceres-solver from apt and I am ok thanks for checking

(base) mona@ard-gpu-01:~$ apt list --installed | grep ceres

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libceres-dev/jammy,now 2.0.0+dfsg1-5 amd64 [installed]
libceres2/jammy,now 2.0.0+dfsg1-5 amd64 [installed,automatic]