search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.24k stars 2.87k forks source link

[Build] nvcc fatal : 'ON': expected a number #19833

Closed adamjstewart closed 4 months ago

adamjstewart commented 6 months ago

Describe the issue

When I build onnxruntime with CUDA support, it tries to use:

nvcc ... --theads ON

which fails with:

nvcc fatal   : 'ON': expected a number

Urgency

Not really, will build without CUDA support for now.

Target platform

linux-ubuntu22.04-zen2

Build script

$ spack install py-onnxruntime+cuda

Error / output

build log

Visual Studio Version

No response

GCC / Compiler Version

12.3.0

sg0771 commented 6 months ago

Windows Build onnxruntime with cuda with the same error

"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvcc.exe" --use-local-env -ccbin "C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.37.32822\bin\HostX64\x86" -x cu -I"C:\work\onnxruntime\win32_deps\utf8_range-src" -IC:\work\onnxruntime\include\onnxruntime -IC:\work\onnxruntime\include\onnxruntime\core\session -I"C:\work\onnxruntime\win32_deps\pytorch_cpuinfo-src\include" -IC:\work\onnxruntime\win32 -IC:\work\onnxruntime\onnxruntime -I"C:\work\onnxruntime\win32_deps\abseil_cpp-src" -I"C:\work\onnxruntime\win32_deps\safeint-src" -I"C:\work\onnxruntime\win32_deps\gsl-src\include" -I"C:\work\onnxruntime\win32_deps\onnx-src" -I"C:\work\onnxruntime\win32_deps\onnx-build" -I"C:\work\onnxruntime\win32_deps\protobuf-src\src" -I"C:\work\onnxruntime\win32_deps\flatbuffers-src\include" -I"C:\work\onnxruntime\win32_deps\cutlass-src\include" -I"C:\work\onnxruntime\win32_deps\cutlass-src\examples" -I"C:\work\onnxruntime\win32_deps\cutlass-src\tools\util\include" -I"C:\work\onnxruntime\win32_deps\eigen-src" -I"C:\work\onnxruntime\win32_deps\mp11-src\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include" -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\include" --keep-dir Debug -maxrregcount=0 --machine 32 --compile -cudart shared -gencode=arch=compute_37,code=sm_37 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe --diag_suppress=bad_friend_decl -Xcudafe --diag_suppress=unsigned_compare_with_zero -Xcudafe --diag_suppress=expr_has_no_effect -std=c++17 -Xcudafe --diag_suppress=conversion_function_not_usable --threads ON -Werror all-warnings -Xcompiler="/EHsc -Ob0 -Zi /utf-8 /sdl /experimental:external /external:W0 /external:templates- /external:IC:/work/onnxruntime/cmake /external:IC:/work/onnxruntime/win32 /wd4251 /wd4201 /wd4324 /wd5054 /w15038 /wd4251 /wd4201 /wd4324 /wd5054 /w15038 /wd4834 /wd4127" -g -D_WINDOWS -DCPUINFO_SUPPORTED_PLATFORM=1 -DENABLE_CPU_FP16_TRAINING_OPS -DEIGEN_USE_THREADS -DDISABLE_CUSPARSE_DEPRECATED -DPLATFORM_WINDOWS -DNOGDI -DNOMINMAX -D_USE_MATH_DEFINES -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS -DUSE_CUDA=1 -DUSE_MEMORY_EFFICIENT_ATTENTION=1 -DONNX_NAMESPACE=onnx -DONNX_ML=1 -DONNX_USE_LITE_PROTO=1 -D__ONNX_NO_DOC_STRINGS -DWIN32_LEAN_AND_MEAN -DORT_ENABLE_STREAM -DEIGEN_MPL2_ONLY -DEIGEN_HAS_CONSTEXPR -DEIGEN_HAS_VARIADIC_TEMPLATES -DEIGEN_HAS_CXX11_MATH -DEIGEN_HAS_CXX11_ATOMIC -DEIGEN_STRONG_INLINE=inline -DENABLE_CUDA_NHWC_OPS -D"CMAKE_INTDIR=\"Debug\"" -Donnxruntime_providers_cuda_EXPORTS -D_WINDLL -D_MBCS -DWIN32 -D_WINDOWS -DEIGEN_HAS_C99_MATH -DCPUINFO_SUPPORTED -DCPUINFO_SUPPORTED_PLATFORM=1 -DENABLE_CPU_FP16_TRAINING_OPS -DEIGEN_USE_THREADS -DDISABLE_CUSPARSE_DEPRECATED -DPLATFORM_WINDOWS -DNOGDI -DNOMINMAX -D_USE_MATH_DEFINES -D_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS -DUSE_CUDA=1 -DUSE_MEMORY_EFFICIENT_ATTENTION=1 -DONNX_NAMESPACE=onnx -DONNX_ML=1 -DONNX_USE_LITE_PROTO=1 -D__ONNX_NO_DOC_STRINGS -DWIN32_LEAN_AND_MEAN -DORT_ENABLE_STREAM -DEIGEN_MPL2_ONLY -DEIGEN_HAS_CONSTEXPR -DEIGEN_HAS_VARIADIC_TEMPLATES -DEIGEN_HAS_CXX11_MATH -DEIGEN_HAS_CXX11_ATOMIC -DEIGEN_STRONG_INLINE=inline -DENABLE_CUDA_NHWC_OPS -D"CMAKE_INTDIR=\"Debug\"" -Donnxruntime_providers_cuda_EXPORTS -Xcompiler "/EHsc /W4 /nologo /Od /Fdonnxruntime_providers_cuda.dir\Debug\vc143.pdb /FS /Zi /RTC1 /MDd /GR" -o onnxruntime_providerscuda.dir\Debug\/C/work/onnxruntime/onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.cu.obj "C:\work\onnxruntime\onnxruntime\contrib_ops\cuda\quantization\matmul_nbits.cu"

jesperheuver commented 4 months ago

Hey,

similar results here when trying to compile onnxruntime from source on a NVidia Jetson AGX Orin with Jetpack 6.0.

Description

The build seems to fail halfway through the build on the cuda provider, giving the message nvcc fatal : 'ON': expected a number I am trying to compile onnxruntime 1.17.3 against cuda 12.2 (preinstalled version on the jetpack 6.0.0 release, as well as tensorrt).

For my application I need to be able to use tensorrt, due to the optimizations available.

Urgency

To me: relatively high. :) Our development would get stuck on this issue.

Target system

Linux ubuntu 5.15.136-tegra #1 SMP PREEMPT Mon May 6 09:56:39 PDT 2024 aarch64 aarch64 aarch64 GNU/Linux Jetpack 6.0.0

# R36 (release), REVISION: 3.0, GCID: 36106755, BOARD: generic, EABI: aarch64, DATE: Thu Apr 25 03:14:05 UTC 2024 \# KERNEL_VARIANT: oot

build script

cmake \
  -Donnxruntime_ENABLE_LANGUAGE_INTEROP_OPS=true \
  -Donnxruntime_ENABLE_CUDA_PROFILING=true \
  -Donnxruntime_USE_CUDA=true \
  -Donnxruntime_BUILD_FOR_NATIVE_MACHINE=true \
  -Donnxruntime_USE_TENSORRT=true \
  -Donnxruntime_BUILD_SHARED_LIB=true \
  -S /opt/onnxruntime/onnxruntime-1.17.3/cmake \
  -B /opt/onnxruntime/onnxruntime-1.17.3/build

compiler version

nvcc --version

Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0

gcc --version

gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Error/output

Terminal output:

[ 51%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/opt/onnxruntime/onnxruntime-1.17.3/onnxruntime/core/providers/cuda/activation/activations_impl.cu.o
nvcc fatal   : 'ON': expected a number
gmake[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:1365: CMakeFiles/onnxruntime_providers_cuda.dir/opt/onnxruntime/onnxruntime-1.17.3/onnxruntime/core/providers/cuda/activation/activations_impl.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:2117: CMakeFiles/onnxruntime_providers_cuda.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

makefile code at the location the crash seems to happen:

/usr/local/cuda-12.2/bin/nvcc -forward-unknown-to-host-compiler $(CUDA_DEFINES) $(CUDA_INCLUDES) $(CUDA_FLAGS) -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/opt/onnxruntime/onnxruntime-1.17.3/onnxruntime/core/providers/cuda/activation/activations_impl.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/opt/onnxruntime/onnxruntime-1.17.3/onnxruntime/core/providers/cuda/activation/activations_impl.cu.o.d -x cu -c /opt/onnxruntime/onnxruntime-1.17.3/onnxruntime/core/providers/cuda/activation/activations_impl.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/opt/onnxruntime/onnxruntime-1.17.3/onnxruntime/core/providers/cuda/activation/activations_impl.cu.o