search

OpenGVLab / Vision-RWKV

Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures
https://arxiv.org/abs/2403.02308
Apache License 2.0
288 stars 11 forks source link

local/cuda/lib64 -lcudart -o wkv6_v1.so Loading extension module wkv6_v1... Aborted (core dumped) #25

Open 45ghost opened 2 weeks ago

45ghost commented 2 weeks ago

您好我想问一下关于Aborted (core dumped)的问题，我一开始还能训练 后来就突然报错这个问题。 我调整了T_MAX参数。也重新配置了环境。都没有解决问题，请问您遇到相同的问题吗？这是怎么引起的 ![Uploading image.png…]()

45ghost commented 2 weeks ago

/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/mmcv/init.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details. warnings.warn( Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv/build.ninja... /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( Building extension module wkv... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output wkv_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -res-usage --maxrregcount 60 --use_fast_math -O3 -Xptxas -O3 -DTmax=256 -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda/wkv_cuda.cu -o wkv_cuda.cuda.o ptxas info : 0 bytes gmem ptxas info : Compiling entry function '_Z15kernel_backwardIfEviiiPKT_S2_S2_S2_S2_PS0_S3_S3S3' for 'sm_89' ptxas info : Function properties for _Z15kernel_backwardIfEviiiPKT_S2_S2_S2_S2_PS0_S3_S3S3 96 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 60 registers, 10240 bytes smem, 440 bytes cmem[0], 16 bytes cmem[2] ptxas info : Compiling entry function '_Z14kernel_forwardIfEviiiPKT_S2_S2_S2PS0' for 'sm_89' ptxas info : Function properties for _Z14kernel_forwardIfEviiiPKT_S2_S2_S2PS0 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 60 registers, 6144 bytes smem, 408 bytes cmem[0], 16 bytes cmem[2] [2/3] c++ -MMD -MF wkv_op.o.d -DTORCH_EXTENSION_NAME=wkv -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda/wkv_op.cpp -o wkv_op.o [3/3] c++ wkv_op.o wkv_cuda.cuda.o -shared -L/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o wkv.so Loading extension module wkv... Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... Creating extension directory /root/.cache/torch_extensions/py310_cu121/wkv6... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv6/build.ninja... /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( Building extension module wkv6... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] c++ -MMD -MF wkv6_op.o.d -DTORCH_EXTENSION_NAME=wkv6 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_op.cpp -o wkv6_op.o [2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output wkv6_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=wkv6 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -res-usage --use_fast_math -O3 -Xptxas -O3 --extra-device-vectorization -DN=64 -DT=256 -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_cuda.cu -o wkv6_cuda.cuda.o ptxas info : 1 bytes gmem ptxas info : Compiling entry function '_Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2PS0' for 'sm_89' ptxas info : Function properties for _Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2PS0 1016 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0] ptxas info : Compiling entry function '_Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2PS0' for 'sm_89' ptxas info : Function properties for _Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2PS0 1016 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0] ptxas info : Compiling entry function '_Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5S5' for 'sm_89' ptxas info : Function properties for _Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5S5 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 244 registers, 1536 bytes smem, 448 bytes cmem[0] ptxas info : Compiling entry function '_Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2PS0' for 'sm_89' ptxas info : Function properties for _Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2PS0 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 100 registers, 1024 bytes smem, 416 bytes cmem[0] [3/3] c++ wkv6_op.o wkv6_cuda.cuda.o -shared -L/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o wkv6.so Loading extension module wkv6... Using /root/.cache/torch_extensions/py310_cu121 as PyTorch extensions root... The input conditions for extension module wkv6 have changed. Bumping to version 1 and re-building as wkv6_v1... Detected CUDA files, patching ldflags Emitting ninja build file /root/.cache/torch_extensions/py310_cu121/wkv6/build.ninja... /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1967: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST']. warnings.warn( Building extension module wkv6_v1... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] c++ -MMD -MF wkv6_op.o.d -DTORCH_EXTENSION_NAME=wkv6_v1 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_op.cpp -o wkv6_op.o [2/3] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output wkv6_cuda.cuda.o.d -DTORCH_EXTENSION_NAME=wkv6_v1 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/TH -isystem /usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/local/miniconda3/envs/RWKV/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_89,code=compute_89 -gencode=arch=compute_89,code=sm_89 --compiler-options '-fPIC' -res-usage --use_fast_math -O3 -Xptxas -O3 --extra-device-vectorization -DN=64 -DT=1024 -std=c++17 -c /hy-tmp/2024HMLIC/RWKV/classification/mmcls_custom/models/backbones/cuda_v6/wkv6_cuda.cu -o wkv6_cuda.cuda.o ptxas info : 1 bytes gmem ptxas info : Compiling entry function '_Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2PS0' for 'sm_89' ptxas info : Function properties for _Z19kernel_backward_333IfEviiiiPKT_S2_S2_PKfS2_S2PS0 4088 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0] ptxas info : Compiling entry function '_Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2PS0' for 'sm_89' ptxas info : Function properties for _Z19kernel_backward_222IfEviiiiPKT_S2_S2_PKfS2_S2PS0 4088 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 96 registers, 512 bytes smem, 424 bytes cmem[0] ptxas info : Compiling entry function '_Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5S5' for 'sm_89' ptxas info : Function properties for _Z19kernel_backward_111IfEviiiiPKT_S2_S2_PKfS2_S2_PS0_S5_S5S5 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 244 registers, 1536 bytes smem, 448 bytes cmem[0] ptxas info : Compiling entry function '_Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2PS0' for 'sm_89' ptxas info : Function properties for _Z14kernel_forwardIfEviiiiPKT_S2_S2_PKfS2PS0 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads ptxas info : Used 100 registers, 1024 bytes smem, 416 bytes cmem[0] [3/3] c++ wkv6_op.o wkv6_cuda.cuda.o -shared -L/usr/local/miniconda3/envs/RWKV/lib/python3.10/site-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o wkv6_v1.so Loading extension module wkv6_v1... Aborted (core dumped)