Open sab148 opened 4 years ago
` $ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130 $ python3 Python 3.7.3 (default, Mar 27 2019, 22:11:17) [GCC 7.3.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.
import torch torch.version.cuda 10.0.130
$ /home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-gcc --version x86_64-conda_cos6-linux-gnu-gcc (crosstool-NG 1.23.0.449-a04d0) 7.3.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ which gcc /usr/local/bin/gcc
$ gcc --version gcc (GCC) 4.9.2 Copyright (C) 2014 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
and also when i tried
CUDA_HOST_COMPILER=~/anaconda3/bin/x86_64-conda_cos6-linux-gnu-gcc python3 setup.py install --cuda_ext --cpp_ext
I got the similar error
modify setup.py:108 torch.utils.cpp_extension.CUDA_HOME = '/usr/local/cuda'.
@sunsolo is right, but it won't work if you're using Anaconda, because in that case you need to refer to the cudatoolkit
installed within your Anaconda environment. Install this version of cudatoolkit (the standard version won't have the nvcc
which is required for apex
installation) and then add torch.utils.cpp_extension.CUDA_HOME = '/home/<user name>/anaconda3/envs/<env_name>/pkgs/cuda-toolkit'
in setup.py
. In my case, that solved the problem.
For more details, see this thread.
https://www.gitmemory.com/issue/NVIDIA/apex/1090/826620315 But when I reset to the commit with 'e5f2f675de1c4e85e772065e550db55b13264265' (15 April), it is no error occur. git checkout e5f2f675de1c4e85e772065e550db55b13264265
When I tried to run
python3 setup.py install --cuda_ext --cpp_ext
I got ` torch.version = 1.2.0
setup.py:46: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies! warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
Compiling cuda extensions with nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2018 NVIDIA Corporation Built on Sat_Aug_25_21:08:01_CDT_2018 Cuda compilation tools, release 10.0, V10.0.130 from /usr/local/cuda/bin
running install running bdist_egg running egg_info writing apex.egg-info/PKG-INFO writing dependency_links to apex.egg-info/dependency_links.txt writing top-level names to apex.egg-info/top_level.txt reading manifest file 'apex.egg-info/SOURCES.txt' writing manifest file 'apex.egg-info/SOURCES.txt' installing library code to build/bdist.linux-x86_64/egg running install_lib running build_py creating build creating build/lib.linux-x86_64-3.7 creating build/lib.linux-x86_64-3.7/apex copying apex/init.py -> build/lib.linux-x86_64-3.7/apex creating build/lib.linux-x86_64-3.7/apex/RNN copying apex/RNN/RNNBackend.py -> build/lib.linux-x86_64-3.7/apex/RNN copying apex/RNN/init.py -> build/lib.linux-x86_64-3.7/apex/RNN copying apex/RNN/cells.py -> build/lib.linux-x86_64-3.7/apex/RNN copying apex/RNN/models.py -> build/lib.linux-x86_64-3.7/apex/RNN creating build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/init.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/version.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/_amp_state.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/_initialize.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/_process_optimizer.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/amp.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/compat.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/frontend.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/handle.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/opt.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/rnn_compat.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/scaler.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/utils.py -> build/lib.linux-x86_64-3.7/apex/amp copying apex/amp/wrap.py -> build/lib.linux-x86_64-3.7/apex/amp creating build/lib.linux-x86_64-3.7/apex/contrib copying apex/contrib/init.py -> build/lib.linux-x86_64-3.7/apex/contrib creating build/lib.linux-x86_64-3.7/apex/fp16_utils copying apex/fp16_utils/init.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils copying apex/fp16_utils/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils copying apex/fp16_utils/fp16util.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils copying apex/fp16_utils/loss_scaler.py -> build/lib.linux-x86_64-3.7/apex/fp16_utils creating build/lib.linux-x86_64-3.7/apex/multi_tensor_apply copying apex/multi_tensor_apply/init.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply copying apex/multi_tensor_apply/multi_tensor_apply.py -> build/lib.linux-x86_64-3.7/apex/multi_tensor_apply creating build/lib.linux-x86_64-3.7/apex/normalization copying apex/normalization/init.py -> build/lib.linux-x86_64-3.7/apex/normalization copying apex/normalization/fused_layer_norm.py -> build/lib.linux-x86_64-3.7/apex/normalization creating build/lib.linux-x86_64-3.7/apex/optimizers copying apex/optimizers/init.py -> build/lib.linux-x86_64-3.7/apex/optimizers copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/optimizers copying apex/optimizers/fused_lamb.py -> build/lib.linux-x86_64-3.7/apex/optimizers copying apex/optimizers/fused_novograd.py -> build/lib.linux-x86_64-3.7/apex/optimizers copying apex/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/optimizers creating build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/LARC.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/init.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/distributed.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/multiproc.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/optimized_sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/optimized_sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/sync_batchnorm.py -> build/lib.linux-x86_64-3.7/apex/parallel copying apex/parallel/sync_batchnorm_kernel.py -> build/lib.linux-x86_64-3.7/apex/parallel creating build/lib.linux-x86_64-3.7/apex/pyprof copying apex/pyprof/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof creating build/lib.linux-x86_64-3.7/apex/reparameterization copying apex/reparameterization/init.py -> build/lib.linux-x86_64-3.7/apex/reparameterization copying apex/reparameterization/reparameterization.py -> build/lib.linux-x86_64-3.7/apex/reparameterization copying apex/reparameterization/weight_norm.py -> build/lib.linux-x86_64-3.7/apex/reparameterization creating build/lib.linux-x86_64-3.7/apex/amp/lists copying apex/amp/lists/init.py -> build/lib.linux-x86_64-3.7/apex/amp/lists copying apex/amp/lists/functional_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists copying apex/amp/lists/tensor_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists copying apex/amp/lists/torch_overrides.py -> build/lib.linux-x86_64-3.7/apex/amp/lists creating build/lib.linux-x86_64-3.7/apex/contrib/groupbn copying apex/contrib/groupbn/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn copying apex/contrib/groupbn/batch_norm.py -> build/lib.linux-x86_64-3.7/apex/contrib/groupbn creating build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/encdec_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/fast_encdec_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/fast_self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/fast_self_multihead_attn_norm_add_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/self_multihead_attn.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn copying apex/contrib/multihead_attn/self_multihead_attn_func.py -> build/lib.linux-x86_64-3.7/apex/contrib/multihead_attn creating build/lib.linux-x86_64-3.7/apex/contrib/optimizers copying apex/contrib/optimizers/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers copying apex/contrib/optimizers/fp16_optimizer.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers copying apex/contrib/optimizers/fused_adam.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers copying apex/contrib/optimizers/fused_sgd.py -> build/lib.linux-x86_64-3.7/apex/contrib/optimizers creating build/lib.linux-x86_64-3.7/apex/contrib/xentropy copying apex/contrib/xentropy/init.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy copying apex/contrib/xentropy/softmax_xentropy.py -> build/lib.linux-x86_64-3.7/apex/contrib/xentropy creating build/lib.linux-x86_64-3.7/apex/pyprof/nvtx copying apex/pyprof/nvtx/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx copying apex/pyprof/nvtx/nvmarker.py -> build/lib.linux-x86_64-3.7/apex/pyprof/nvtx creating build/lib.linux-x86_64-3.7/apex/pyprof/parse copying apex/pyprof/parse/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse copying apex/pyprof/parse/main.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse copying apex/pyprof/parse/db.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse copying apex/pyprof/parse/kernel.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse copying apex/pyprof/parse/nvvp.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse copying apex/pyprof/parse/parse.py -> build/lib.linux-x86_64-3.7/apex/pyprof/parse creating build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/init.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/main.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/activation.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/base.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/blas.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/conv.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/convert.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/data.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/dropout.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/embedding.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/index_slice_join_mutate.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/linear.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/loss.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/misc.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/normalization.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/optim.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/output.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/pointwise.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/pooling.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/prof.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/randomSample.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/recurrentCell.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/reduction.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/softmax.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/usage.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof copying apex/pyprof/prof/utility.py -> build/lib.linux-x86_64-3.7/apex/pyprof/prof running build_ext building 'apex_C' extension creating build/temp.linux-x86_64-3.7 creating build/temp.linux-x86_64-3.7/csrc /home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include -fPIC -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/home/narimene/anaconda3/include/python3.7m -c csrc/flatten_unflatten.cpp -o build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ /home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-c++ -pthread -shared -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/home/narimene/anaconda3/lib -Wl,-rpath-link,/home/narimene/anaconda3/lib -L/home/narimene/anaconda3/lib -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include build/temp.linux-x86_64-3.7/csrc/flatten_unflatten.o -o build/lib.linux-x86_64-3.7/apex_C.cpython-37m-x86_64-linux-gnu.so building 'amp_C' extension /home/narimene/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/narimene/anaconda3/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/narimene/anaconda3/include -fPIC -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/narimene/anaconda3/include/python3.7m -c csrc/amp_C_frontend.cpp -o build/temp.linux-x86_64-3.7/csrc/amp_C_frontend.o -O3 -DVERSION_GE_1_1 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ /usr/local/cuda/bin/nvcc -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/TH -I/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda/include -I/home/narimene/anaconda3/include/python3.7m -c csrc/multi_tensor_sgd_kernel.cu -o build/temp.linux-x86_64-3.7/csrc/multi_tensor_sgd_kernel.o -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 /home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/core/TensorTypeIdRegistration.h(50): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/typeid.h(596): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/typeid.h(596): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/c10/util/Registry.h(157): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(23): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(24): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(26): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(30): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/LegacyTypeDispatch.h(34): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/Generator.h(75): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/core/ATenDispatch.h(104): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(114): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(115): error: namespace "std" has no member "once_flag"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(69): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(75): error: namespace "std" has no member "call_once"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(207): error: namespace "std" has no member "mutex"
/home/narimene/anaconda3/lib/python3.7/site-packages/torch/include/ATen/Context.h(218): error: namespace "std" has no member "mutex"
17 errors detected in the compilation of "/tmp/tmpxft_0000624a_00000000-6_multi_tensor_sgd_kernel.cpp1.ii". error: command '/usr/local/cuda/bin/nvcc' failed with exit status 1 `
Please if someone can help me