search

PaddlePaddle / Paddle

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）
http://www.paddlepaddle.org/
Apache License 2.0
22.22k stars 5.58k forks source link

windows下paddle 2.4、cuda 11.6源码编译失败 #48542

Closed sdcb closed 10 months ago

sdcb commented 1 year ago

bug描述 Describe the Bug

分支：release/2.4

cmake configure命令：

set vcvars64_dir="C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat"
call %vcvars64_dir%

SET CUDA_TOOLKIT_ROOT_DIR=C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.6
SET PATH=%CUDA_TOOLKIT_ROOT_DIR%/bin;%CUDA_TOOLKIT_ROOT_DIR%/libnvvp;%PATH%
cmake .. -GNinja -DWITH_MKL=ON -DWITH_GPU=ON -DON_INFER=ON -DWITH_PYTHON=OFF -DWITH_UNITY_BUILD=ON -DCUDA_ARCH_NAME=Manual -DCUDA_ARCH_BIN="61 75 86" -DCMAKE_CUDA_ARCHITECTURES="61-real;75-real;86-real" -DWITH_ONNXRUNTIME=OFF -DWITH_TENSORRT=OFF -DCUDA_TOOLKIT_ROOT_DIR="%CUDA_TOOLKIT_ROOT_DIR%"

ninja all

报错内容：

[87/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj
FAILED: paddle/phi/kernels/CMakeFiles/phi_gpu.dir/gpu/logcumsumexp_grad_kernel.cu.obj
C:\PROGRA~1\NVIDIA~2\CUDA\v11.6\bin\nvcc.exe -forward-unknown-to-host-compiler -DCUDA_TOOLKIT_ROOT_DIR="\"C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.6\"" -DCUDA_VERSION_MAJOR=\"11\" -DCUDA_VERSION_MINOR=\"6\" -DCUDNN_MAJOR_VERSION=\"8\" -DEIGEN_STRONG_INLINE=inline -DEIGEN_USE_GPU -DGOOGLE_GLOG_DLL_DECL="" -DLAPACK_FOUND -DNOMINMAX -DPADDLE_DISABLE_PROFILER -DPADDLE_DLL_EXPORT -DPADDLE_DLL_INFERENCE -DPADDLE_NO_PYTHON -DPADDLE_ON_INFERENCE -DPADDLE_VERSION=0.0.0 -DPADDLE_VERSION_INTEGER=0 -DPADDLE_WITH_AVX -DPADDLE_WITH_CRYPTO -DPADDLE_WITH_CUDA -DPADDLE_WITH_MKLDNN -DPADDLE_WITH_MKLML -DPADDLE_WITH_POCKETFFT -DPADDLE_WITH_SSE3 -DTHRUST_IGNORE_CUB_VERSION_CHECK -DTRT_PLUGIN_FP16_AVALIABLE -DUTF8PROC_STATIC -D_XKEYCHECK_H -IC:\_\code\Paddle\build -IC:\_\code\Paddle\paddle\fluid\framework\io -IC:\_\code\Paddle\build\third_party\install\zlib\include -IC:\_\code\Paddle\build\third_party\install -IC:\_\code\Paddle\build\third_party\install\gflags\include -IC:\_\code\Paddle\build\third_party\install\glog\include -IC:\_\code\Paddle\build\third_party\eigen3\src\extern_eigen3 -IC:\_\code\Paddle\build\third_party\threadpool\src\extern_threadpool -IC:\_\code\Paddle\build\third_party\dlpack\src\extern_dlpack\include -IC:\_\code\Paddle\build\third_party\install\xxhash\include -IC:\_\code\Paddle\build\third_party\install\warpctc\include -IC:\_\code\Paddle\build\third_party\install\utf8proc\include -IC:\_\code\Paddle\build\third_party\install\mklml\include -IC:\_\code\Paddle\build\third_party\install\mkldnn\include -IC:\_\code\Paddle\build\third_party\install\protobuf\include -IC:\_\code\Paddle\build\third_party\cub\src\extern_cub -IC:\_\code\Paddle\build\third_party\install\cryptopp\include -IC:\_\code\Paddle\build\third_party\pocketfft\src -IC:\_\code\Paddle\build\third_party\dirent\src\extern_dirent\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IC:\_\code\Paddle -IC:\_\code\Paddle\build\..\paddle\fluid\framework\io -D_WINDOWS -Xcompiler=" /W0  /GR /EHsc" -D_MWAITXINTRIN_H_INCLUDED -D__STRICT_ANSI__ -Wno-deprecated-gpu-targets  -gencode arch=compute_61,code=sm_61 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_86,code=sm_86 -w --expt-relaxed-constexpr --expt-extended-lambda -Xcompiler "/wd4244 /wd4267 /wd4819 " -Xcompiler /bigobj  -Xcompiler="/arch:AVX" -Xcompiler="-MT -O2 -Ob2" -DNDEBUG --generate-code=arch=compute_61,code=[sm_61] --generate-code=arch=compute_75,code=[sm_75] --generate-code=arch=compute_86,code=[sm_86] -std=c++14 -MD -MT paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj -MF paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj.d -x cu -c C:\_\code\Paddle\paddle\phi\kernels\gpu\logcumsumexp_grad_kernel.cu -o paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj -Xcompiler=-Fdpaddle\phi\kernels\CMakeFiles\phi_gpu.dir\phi_gpu.pdb,-FS
logcumsumexp_grad_kernel.cu
INTERNAL COMPILER ERROR in 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe'
    Please choose the Technical Support command on the Visual C++
    Help menu, or open the Technical Support help file for more information
nvcc error   : 'cl.exe' died with status 0xC0000005 (ACCESS_VIOLATION)

报错截图： image

详细完整日志：

C:\_\code\Paddle\build>ninja all
[6/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fill_diagonal_tensor_kernel.cu.obj
fill_diagonal_tensor_kernel.cu
[7/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fill_diagonal_tensor_grad_kernel.cu.obj
fill_diagonal_tensor_grad_kernel.cu
[8/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\expand_as_grad_kernel.cu.obj
expand_as_grad_kernel.cu
[9/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\expand_grad_kernel.cu.obj
expand_grad_kernel.cu
[10/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\erfinv_grad_kernel.cu.obj
erfinv_grad_kernel.cu
[11/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fill_diagonal_grad_kernel.cu.obj
fill_diagonal_grad_kernel.cu
[12/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fill_diagonal_kernel.cu.obj
fill_diagonal_kernel.cu
[13/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\exponential_kernel.cu.obj
exponential_kernel.cu
[14/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fft_kernel.cu.obj
fft_kernel.cu
[15/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\expand_as_kernel.cu.obj
expand_as_kernel.cu
[16/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fft_grad_kernel.cu.obj
fft_grad_kernel.cu
[17/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\eye_kernel.cu.obj
eye_kernel.cu
[18/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fill_grad_kernel.cu.obj
fill_grad_kernel.cu
[19/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fill_kernel.cu.obj
fill_kernel.cu
[20/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\eigh_grad_kernel.cu.obj
eigh_grad_kernel.cu
[21/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\expand_kernel.cu.obj
expand_kernel.cu
[22/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\erfinv_kernel.cu.obj
erfinv_kernel.cu
[23/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\flip_kernel.cu.obj
flip_kernel.cu
[24/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gather_tree_kernel.cu.obj
gather_tree_kernel.cu
[25/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fold_kernel.cu.obj
fold_kernel.cu
[26/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\fold_grad_kernel.cu.obj
fold_grad_kernel.cu
[27/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\frame_grad_kernel.cu.obj
frame_grad_kernel.cu
[28/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\frame_kernel.cu.obj
frame_kernel.cu
[29/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gather_nd_grad_kernel.cu.obj
gather_nd_grad_kernel.cu
[30/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\frobenius_norm_grad_kernel.cu.obj
frobenius_norm_grad_kernel.cu
[31/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gather_kernel.cu.obj
gather_kernel.cu
[32/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\full_kernel.cu.obj
full_kernel.cu
[33/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gather_grad_kernel.cu.obj
gather_grad_kernel.cu
[34/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gaussian_random_kernel.cu.obj
gaussian_random_kernel.cu
[35/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gather_nd_kernel.cu.obj
gather_nd_kernel.cu
[36/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_reindex_kernel.cu.obj
graph_reindex_kernel.cu
[37/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gelu_kernel.cu.obj
gelu_kernel.cu
[38/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_send_recv_grad_kernel.cu.obj
graph_send_recv_grad_kernel.cu
[39/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gelu_grad_kernel.cu.obj
gelu_grad_kernel.cu
[40/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_send_recv_kernel.cu.obj
graph_send_recv_kernel.cu
[41/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_sample_neighbors_kernel.cu.obj
graph_sample_neighbors_kernel.cu
[42/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\grid_sample_kernel.cu.obj
grid_sample_kernel.cu
[43/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\generate_proposals_v2_kernel.cu.obj
generate_proposals_v2_kernel.cu
[44/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_send_ue_recv_kernel.cu.obj
graph_send_ue_recv_kernel.cu
[45/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_send_uv_kernel.cu.obj
graph_send_uv_kernel.cu
[46/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_send_ue_recv_grad_kernel.cu.obj
graph_send_ue_recv_grad_kernel.cu
[47/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\frobenius_norm_kernel.cu.obj
frobenius_norm_kernel.cu
[48/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\huber_loss_grad_kernel.cu.obj
huber_loss_grad_kernel.cu
[49/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gumbel_softmax_grad_kernel.cu.obj
gumbel_softmax_grad_kernel.cu
[50/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\grid_sample_grad_kernel.cu.obj
grid_sample_grad_kernel.cu
[51/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\group_norm_grad_kernel.cu.obj
group_norm_grad_kernel.cu
[52/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\huber_loss_kernel.cu.obj
huber_loss_kernel.cu
[53/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\graph_send_uv_grad_kernel.cu.obj
graph_send_uv_grad_kernel.cu
[54/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\increment_kernel.cu.obj
increment_kernel.cu
[55/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\group_norm_kernel.cu.obj
group_norm_kernel.cu
[56/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\histogram_kernel.cu.obj
histogram_kernel.cu
[57/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\index_add_grad_kernel.cu.obj
index_add_grad_kernel.cu
[58/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\index_add_kernel.cu.obj
index_add_kernel.cu
[59/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\isclose_kernel.cu.obj
isclose_kernel.cu
[60/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\index_select_kernel.cu.obj
index_select_kernel.cu
[61/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\index_sample_grad_kernel.cu.obj
index_sample_grad_kernel.cu
[62/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\elementwise_subtract_grad_kernel.cu.obj
elementwise_subtract_grad_kernel.cu
[63/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\inverse_kernel.cu.obj
inverse_kernel.cu
[64/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\gumbel_softmax_kernel.cu.obj
gumbel_softmax_kernel.cu
[65/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\index_sample_kernel.cu.obj
index_sample_kernel.cu
[66/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\kldiv_loss_grad_kernel.cu.obj
kldiv_loss_grad_kernel.cu
[67/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\index_select_grad_kernel.cu.obj
index_select_grad_kernel.cu
[68/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\kldiv_loss_kernel.cu.obj
kldiv_loss_kernel.cu
[69/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\instance_norm_grad_kernel.cu.obj
instance_norm_grad_kernel.cu
[70/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\interpolate_kernel.cu.obj
interpolate_kernel.cu
[71/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\instance_norm_kernel.cu.obj
instance_norm_kernel.cu
[72/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\interpolate_grad_kernel.cu.obj
interpolate_grad_kernel.cu
[73/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\isfinite_kernel.cu.obj
isfinite_kernel.cu
[74/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\inverse_grad_kernel.cu.obj
inverse_grad_kernel.cu
[75/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\kron_kernel.cu.obj
kron_kernel.cu
[76/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\label_smooth_grad_kernel.cu.obj
label_smooth_grad_kernel.cu
[77/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lerp_kernel.cu.obj
lerp_kernel.cu
[78/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\log_loss_grad_kernel.cu.obj
log_loss_grad_kernel.cu
[79/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lgamma_grad_kernel.cu.obj
lgamma_grad_kernel.cu
[80/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\kthvalue_grad_kernel.cu.obj
kthvalue_grad_kernel.cu
[81/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\log_loss_kernel.cu.obj
log_loss_kernel.cu
[82/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\label_smooth_kernel.cu.obj
label_smooth_kernel.cu
[83/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\linspace_kernel.cu.obj
linspace_kernel.cu
[84/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lamb_kernel.cu.obj
lamb_kernel.cu
[85/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lgamma_kernel.cu.obj
lgamma_kernel.cu
[86/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\kthvalue_kernel.cu.obj
kthvalue_kernel.cu
[87/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj
FAILED: paddle/phi/kernels/CMakeFiles/phi_gpu.dir/gpu/logcumsumexp_grad_kernel.cu.obj
C:\PROGRA~1\NVIDIA~2\CUDA\v11.6\bin\nvcc.exe -forward-unknown-to-host-compiler -DCUDA_TOOLKIT_ROOT_DIR="\"C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.6\"" -DCUDA_VERSION_MAJOR=\"11\" -DCUDA_VERSION_MINOR=\"6\" -DCUDNN_MAJOR_VERSION=\"8\" -DEIGEN_STRONG_INLINE=inline -DEIGEN_USE_GPU -DGOOGLE_GLOG_DLL_DECL="" -DLAPACK_FOUND -DNOMINMAX -DPADDLE_DISABLE_PROFILER -DPADDLE_DLL_EXPORT -DPADDLE_DLL_INFERENCE -DPADDLE_NO_PYTHON -DPADDLE_ON_INFERENCE -DPADDLE_VERSION=0.0.0 -DPADDLE_VERSION_INTEGER=0 -DPADDLE_WITH_AVX -DPADDLE_WITH_CRYPTO -DPADDLE_WITH_CUDA -DPADDLE_WITH_MKLDNN -DPADDLE_WITH_MKLML -DPADDLE_WITH_POCKETFFT -DPADDLE_WITH_SSE3 -DTHRUST_IGNORE_CUB_VERSION_CHECK -DTRT_PLUGIN_FP16_AVALIABLE -DUTF8PROC_STATIC -D_XKEYCHECK_H -IC:\_\code\Paddle\build -IC:\_\code\Paddle\paddle\fluid\framework\io -IC:\_\code\Paddle\build\third_party\install\zlib\include -IC:\_\code\Paddle\build\third_party\install -IC:\_\code\Paddle\build\third_party\install\gflags\include -IC:\_\code\Paddle\build\third_party\install\glog\include -IC:\_\code\Paddle\build\third_party\eigen3\src\extern_eigen3 -IC:\_\code\Paddle\build\third_party\threadpool\src\extern_threadpool -IC:\_\code\Paddle\build\third_party\dlpack\src\extern_dlpack\include -IC:\_\code\Paddle\build\third_party\install\xxhash\include -IC:\_\code\Paddle\build\third_party\install\warpctc\include -IC:\_\code\Paddle\build\third_party\install\utf8proc\include -IC:\_\code\Paddle\build\third_party\install\mklml\include -IC:\_\code\Paddle\build\third_party\install\mkldnn\include -IC:\_\code\Paddle\build\third_party\install\protobuf\include -IC:\_\code\Paddle\build\third_party\cub\src\extern_cub -IC:\_\code\Paddle\build\third_party\install\cryptopp\include -IC:\_\code\Paddle\build\third_party\pocketfft\src -IC:\_\code\Paddle\build\third_party\dirent\src\extern_dirent\include -I"C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.6\include" -IC:\_\code\Paddle -IC:\_\code\Paddle\build\..\paddle\fluid\framework\io -D_WINDOWS -Xcompiler=" /W0  /GR /EHsc" -D_MWAITXINTRIN_H_INCLUDED -D__STRICT_ANSI__ -Wno-deprecated-gpu-targets  -gencode arch=compute_61,code=sm_61 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_86,code=sm_86 -w --expt-relaxed-constexpr --expt-extended-lambda -Xcompiler "/wd4244 /wd4267 /wd4819 " -Xcompiler /bigobj  -Xcompiler="/arch:AVX" -Xcompiler="-MT -O2 -Ob2" -DNDEBUG --generate-code=arch=compute_61,code=[sm_61] --generate-code=arch=compute_75,code=[sm_75] --generate-code=arch=compute_86,code=[sm_86] -std=c++14 -MD -MT paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj -MF paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj.d -x cu -c C:\_\code\Paddle\paddle\phi\kernels\gpu\logcumsumexp_grad_kernel.cu -o paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logcumsumexp_grad_kernel.cu.obj -Xcompiler=-Fdpaddle\phi\kernels\CMakeFiles\phi_gpu.dir\phi_gpu.pdb,-FS
logcumsumexp_grad_kernel.cu
INTERNAL COMPILER ERROR in 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX64\x64\cl.exe'
    Please choose the Technical Support command on the Visual C++
    Help menu, or open the Technical Support help file for more information
nvcc error   : 'cl.exe' died with status 0xC0000005 (ACCESS_VIOLATION)
[88/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\log_softmax_grad_kernel.cu.obj
log_softmax_grad_kernel.cu
[89/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\kron_grad_kernel.cu.obj
kron_grad_kernel.cu
[90/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logspace_kernel.cu.obj
logspace_kernel.cu
[91/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\log_softmax_kernel.cu.obj
log_softmax_kernel.cu
[92/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logsumexp_grad_kernel.cu.obj
logsumexp_grad_kernel.cu
[93/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\layer_norm_kernel.cu.obj
layer_norm_kernel.cu
[94/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lstsq_kernel.cu.obj
lstsq_kernel.cu
[95/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lu_kernel.cu.obj
lu_kernel.cu
[96/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\margin_cross_entropy_grad_kernel.cu.obj
margin_cross_entropy_grad_kernel.cu
[97/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\masked_select_grad_kernel.cu.obj
masked_select_grad_kernel.cu
[98/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lu_unpack_grad_kernel.cu.obj
lu_unpack_grad_kernel.cu
[99/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lu_unpack_kernel.cu.obj
lu_unpack_kernel.cu
[100/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lu_grad_kernel.cu.obj
lu_grad_kernel.cu
[101/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\lerp_grad_kernel.cu.obj
lerp_grad_kernel.cu
[102/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\margin_cross_entropy_kernel.cu.obj
margin_cross_entropy_kernel.cu
[103/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\layer_norm_grad_kernel.cu.obj
layer_norm_grad_kernel.cu
[104/1662] Building CUDA object paddle\phi\kernels\CMakeFiles\phi_gpu.dir\gpu\logsumexp_kernel.cu.obj
logsumexp_kernel.cu
ninja: build stopped: subcommand failed.

C:\_\code\Paddle\build>

其他补充信息 Additional Supplementary Information

似乎每次ninja all报错的***.cu文件都不同，这次是logcumsumexp_grad_kernel.cu，上次是elementwise_subtract_grad_kernel.cu

paddle-bot[bot] commented 1 year ago

您好，我们已经收到了您的问题，会安排技术人员尽快解答您的问题，请耐心等待。请您再次检查是否提供了清晰的问题描述、复现代码、环境&版本、报错信息等。同时，您也可以通过查看官网API文档、常见问题、历史Issue、AI社区来寻求解答。祝您生活愉快～

Hi! We've received your issue and please be patient to get responded. We will arrange technicians to answer your questions as soon as possible. Please make sure that you have posted enough message to demo your request. You may also check out the API，FAQ，Github Issue and AI community to get the answer.Have a nice day!

cuicheng01 commented 1 year ago

您的问题已收到，我们看下问题哈～

zhwesky2010 commented 1 year ago

@sdcb 你好，看起来是VS的访问权限不足，是否是管理员账户

sdcb commented 1 year ago

@zhouwei25 你好，我确实是管理员帐户Administrator，为何会觉得是权限不足？

sdcb commented 1 year ago

@zhouwei25 经过不停编译、不停重试，最终竟然编译成功了，本质这个问题是偶发性的，但我试过使用CUDA 10.2编译出现问题的机率小许多，为什么会有这种偶发性问题呢？

zhwesky2010 commented 1 year ago

@sdcb 因为这里报错的是 'cl.exe' died with status 0xC0000005 (ACCESS_VIOLATION) ，可能和操作系统权限有关

paddle-bot[bot] commented 10 months ago

Since you haven\'t replied for more than a year, we have closed this issue/pr. If the problem is not solved or there is a follow-up one, please reopen it at any time and we will continue to follow up. 由于您超过一年未回复，我们将关闭这个issue/pr。 若问题未解决或有后续问题，请随时重新打开，我们会继续跟进。