uoguelph-mlrg / theano_alexnet

Theano-based Alexnet
BSD 3-Clause "New" or "Revised" License
229 stars 113 forks source link

RuntimeError "Could not compile cuda_convnet" #1

Closed ibogun closed 9 years ago

ibogun commented 9 years ago

I followed instructions, but couldn't run train.py. Here is my error message:

In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1761:0, from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17, from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:4, from /usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/cuda_ndarray.cuh:35, from /home/ivan/Programs/Library/pylearn2/pylearn2/sandbox/cuda_convnet/nvmatrix.cuh:49, from mod.cu:130: /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]

warning "Using deprecated NumPy API, disable it by " \

^ /usr/local/cuda-6.5/bin/..//include/cublas.h(90): error: more than one instance of overloaded function "cublasGetVersion_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(102): error: more than one instance of overloaded function "cublasSnrm2_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(103): error: more than one instance of overloaded function "cublasDnrm2_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(104): error: more than one instance of overloaded function "cublasScnrm2_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(105): error: more than one instance of overloaded function "cublasDznrm2_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(108): error: more than one instance of overloaded function "cublasSdot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(110): error: more than one instance of overloaded function "cublasDdot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(112): error: more than one instance of overloaded function "cublasCdotu_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(114): error: more than one instance of overloaded function "cublasCdotc_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(116): error: more than one instance of overloaded function "cublasZdotu_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(118): error: more than one instance of overloaded function "cublasZdotc_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(122): error: more than one instance of overloaded function "cublasSscal_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(123): error: more than one instance of overloaded function "cublasDscal_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(124): error: more than one instance of overloaded function "cublasCscal_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(125): error: more than one instance of overloaded function "cublasZscal_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(127): error: more than one instance of overloaded function "cublasCsscal_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(128): error: more than one instance of overloaded function "cublasZdscal_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(131): error: more than one instance of overloaded function "cublasSaxpy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(133): error: more than one instance of overloaded function "cublasDaxpy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(135): error: more than one instance of overloaded function "cublasCaxpy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(137): error: more than one instance of overloaded function "cublasZaxpy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(141): error: more than one instance of overloaded function "cublasScopy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(143): error: more than one instance of overloaded function "cublasDcopy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(145): error: more than one instance of overloaded function "cublasCcopy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(147): error: more than one instance of overloaded function "cublasZcopy_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(151): error: more than one instance of overloaded function "cublasSswap_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(152): error: more than one instance of overloaded function "cublasDswap_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(153): error: more than one instance of overloaded function "cublasCswap_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(154): error: more than one instance of overloaded function "cublasZswap_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(157): error: more than one instance of overloaded function "cublasIsamax_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(158): error: more than one instance of overloaded function "cublasIdamax_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(159): error: more than one instance of overloaded function "cublasIcamax_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(160): error: more than one instance of overloaded function "cublasIzamax_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(163): error: more than one instance of overloaded function "cublasIsamin_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(164): error: more than one instance of overloaded function "cublasIdamin_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(166): error: more than one instance of overloaded function "cublasIcamin_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(167): error: more than one instance of overloaded function "cublasIzamin_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(170): error: more than one instance of overloaded function "cublasSasum_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(171): error: more than one instance of overloaded function "cublasDasum_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(172): error: more than one instance of overloaded function "cublasScasum_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(173): error: more than one instance of overloaded function "cublasDzasum_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(176): error: more than one instance of overloaded function "cublasSrot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(178): error: more than one instance of overloaded function "cublasDrot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(180): error: more than one instance of overloaded function "cublasCrot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(182): error: more than one instance of overloaded function "cublasZrot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(185): error: more than one instance of overloaded function "cublasCsrot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(187): error: more than one instance of overloaded function "cublasZdrot_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(191): error: more than one instance of overloaded function "cublasSrotg_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(192): error: more than one instance of overloaded function "cublasDrotg_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(193): error: more than one instance of overloaded function "cublasCrotg_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(195): error: more than one instance of overloaded function "cublasZrotg_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(199): error: more than one instance of overloaded function "cublasSrotm_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(201): error: more than one instance of overloaded function "cublasDrotm_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(205): error: more than one instance of overloaded function "cublasSrotmg_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(207): error: more than one instance of overloaded function "cublasDrotmg_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(212): error: more than one instance of overloaded function "cublasSgemv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(215): error: more than one instance of overloaded function "cublasDgemv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(218): error: more than one instance of overloaded function "cublasCgemv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(221): error: more than one instance of overloaded function "cublasZgemv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(226): error: more than one instance of overloaded function "cublasSgbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(230): error: more than one instance of overloaded function "cublasDgbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(234): error: more than one instance of overloaded function "cublasCgbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(238): error: more than one instance of overloaded function "cublasZgbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(244): error: more than one instance of overloaded function "cublasStrmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(246): error: more than one instance of overloaded function "cublasDtrmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(248): error: more than one instance of overloaded function "cublasCtrmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(250): error: more than one instance of overloaded function "cublasZtrmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(254): error: more than one instance of overloaded function "cublasStbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(256): error: more than one instance of overloaded function "cublasDtbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(258): error: more than one instance of overloaded function "cublasCtbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(260): error: more than one instance of overloaded function "cublasZtbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(264): error: more than one instance of overloaded function "cublasStpmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(266): error: more than one instance of overloaded function "cublasDtpmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(268): error: more than one instance of overloaded function "cublasCtpmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(270): error: more than one instance of overloaded function "cublasZtpmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(273): error: more than one instance of overloaded function "cublasStrsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(275): error: more than one instance of overloaded function "cublasDtrsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(277): error: more than one instance of overloaded function "cublasCtrsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(279): error: more than one instance of overloaded function "cublasZtrsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(283): error: more than one instance of overloaded function "cublasStpsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(286): error: more than one instance of overloaded function "cublasDtpsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(288): error: more than one instance of overloaded function "cublasCtpsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(290): error: more than one instance of overloaded function "cublasZtpsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(294): error: more than one instance of overloaded function "cublasStbsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(298): error: more than one instance of overloaded function "cublasDtbsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(301): error: more than one instance of overloaded function "cublasCtbsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(305): error: more than one instance of overloaded function "cublasZtbsv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(310): error: more than one instance of overloaded function "cublasSsymv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(313): error: more than one instance of overloaded function "cublasDsymv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(316): error: more than one instance of overloaded function "cublasChemv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(319): error: more than one instance of overloaded function "cublasZhemv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(324): error: more than one instance of overloaded function "cublasSsbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(327): error: more than one instance of overloaded function "cublasDsbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(330): error: more than one instance of overloaded function "cublasChbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(333): error: more than one instance of overloaded function "cublasZhbmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(338): error: more than one instance of overloaded function "cublasSspmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(341): error: more than one instance of overloaded function "cublasDspmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(344): error: more than one instance of overloaded function "cublasChpmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(347): error: more than one instance of overloaded function "cublasZhpmv_v2" has "C" linkage /usr/local/cuda-6.5/bin/..//include/cublas.h(353): error: more than one instance of overloaded function "cublasSger_v2" has "C" linkage Error limit reached. 100 errors detected in the compilation of "/tmp/tmpxft_000009c9_00000000-8_mod.cpp1.ii". Compilation terminated.

['nvcc', '-shared', '-g', '-O3', '-use_fast_math', '-arch=sm_30', '-m64', '-Xcompiler', '-DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a70a88a837011,-fPIC', '-Xlinker', '-rpath,/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray', '-Xlinker', '-rpath,/usr/local/cuda-6.5/lib', '-Xlinker', '-rpath,/usr/local/cuda-6.5/lib64', '-I/home/ivan/Programs/Library/pylearn2/pylearn2/sandbox/cuda_convnet/', '-I/usr/lib/python2.7/dist-packages/numpy/core/include', '-I/usr/include/python2.7', '-I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda', '-o', '/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_convnet/cuda_convnet.so', 'mod.cu', '-L/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray', '-L/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_convnet', '-L/usr/local/cuda-6.5/lib', '-L/usr/local/cuda-6.5/lib64', '-L/usr/lib', '-lpython2.7', '-lcublas', '-lcudart'] ERROR (pylearn2.sandbox.cuda_convnet.convnet_compile): Failed to compile /home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_convnet/mod.cu ('nvmatrix_kernels.cu', 'nvmatrix.cu', 'conv_util.cu', 'filter_acts.cu', 'img_acts.cu', 'weight_acts.cu'): ('nvcc return status', 4, 'for cmd', 'nvcc -shared -g -O3 -use_fast_math -arch=sm_30 -m64 -Xcompiler -DCUDA_NDARRAY_CUH=d67f7c8a21306c67152a70a88a837011,-fPIC -Xlinker -rpath,/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray -Xlinker -rpath,/usr/local/cuda-6.5/lib -Xlinker -rpath,/usr/local/cuda-6.5/lib64 -I/home/ivan/Programs/Library/pylearn2/pylearn2/sandbox/cuda_convnet/ -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda -o /home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_convnet/cuda_convnet.so mod.cu -L/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_ndarray -L/home/ivan/.theano/compiledir_Linux-3.13.0-44-generic-x86_64-with-Ubuntu-14.04-trusty-x86_64-2.7.6-64/cuda_convnet -L/usr/local/cuda-6.5/lib -L/usr/local/cuda-6.5/lib64 -L/usr/lib -lpython2.7 -lcublas -lcudart') Process Process-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(_self._args, *_self._kwargs) File "/home/ivan/Code/saliency/python/deep_object/theano_alexnet-master/train.py", line 56, in train_net shared_x, shared_y, rand_arr, vels) = compile_models(model, config) File "/home/ivan/Code/saliency/python/deep_object/theano_alexnet-master/alex_net.py", line 191, in compile_models (rand, rand_arr)]) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 223, in function profile=profile) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 512, in pfunc on_unused_input=on_unused_input) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1312, in orig_function defaults) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1181, in create _fn, _i, _o = self.linker.make_thunk(input_storage=input_storage_lists) File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 434, in make_thunk output_storage=output_storage)[:3] File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 847, in make_all no_recycling)) File "/home/ivan/Programs/Library/pylearn2/pylearn2/sandbox/cuda_convnet/base_acts.py", line 152, in make_thunk raise RuntimeError('Could not compile cuda_convnet') RuntimeError: ('The following error happened while compiling the node', <pylearn2.sandbox.cuda_convnet.filter_acts.FilterActs object at 0x7f009a2df350>(GpuContiguous.0, GpuContiguous.0), '\n', 'Could not compile cuda_convnet')

I am not sure where to go from here. Any ideas?

gwding commented 9 years ago

hi, it seems that there's a problem of theano installation. According to posts in https://groups.google.com/forum/#!msg/pylearn-users/x3jA1pQIN38/ehNH20ZAiuEJ reinstallation of the latest theano would solve this problem.

ibogun commented 9 years ago

Thank you sir, worked like a charm.