theano error

[Abigale001]

I run this command: $python run.py /path/to/training_data_file -t /path/to/test_data_file -m 1 5 10 20 -ps loss=bpr-max,final_act=elu-.5,hidden_act=tanh,layers=100,adapt=adagrad,n_epochs=10,batch_size=32,dropout_p_embed=0.0,dropout_p_hidden=0.0,learning_rate=0.2,momentum=0.3,n_sample=2048,sample_alpha=0.0,bpreg=1.0,constrained_embedding=False

But I get this error:

/home/.../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gpuarray/dnn.py:184: UserWarning: Your cuDNN version is more recent than Theano. If you encounter problems, try updating Theano or downgrading cuDNN to a version >= v5 and <= v7. warnings.warn("Your cuDNN version is more recent than " ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "/home/.../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gpuarray/init .py", line 227, in use(config.device) File "/home/.../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gpuarray/init .py", line 214, in use init_dev(device, preallocate=preallocate) File "/home/.../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gpuarray/_init _.py", line 117, in init_dev context.cudnn_handle = dnn._make_handle(context) File "/home/.../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gpuarray/dnn.py" , line 130, in _make_handle "This can be a sign of a too old driver.", err) RuntimeError: ('Error creating cudnn handle. This can be a sign of a too old driver.', 1) SET loss TO bpr-max (type: <class 'str'>) SET final_act TO elu-0.5 (type: <class 'str'>) SET hidden_act TO tanh (type: <class 'str'>) SET layers TO [100] (type: <class 'list'>) SET adapt TO adagrad (type: <class 'str'>) SET n_epochs TO 10 (type: <class 'int'>) SET batch_size TO 32 (type: <class 'int'>) SET dropout_p_embed TO 0.0 (type: <class 'float'>) SET dropout_p_hidden TO 0.0 (type: <class 'float'>) SET learning_rate TO 0.2 (type: <class 'float'>) SET momentum TO 0.3 (type: <class 'float'>) SET n_sample TO 2048 (type: <class 'int'>) SET sample_alpha TO 0.0 (type: <class 'float'>) SET bpreg TO 1.0 (type: <class 'float'>) SET constrained_embedding TO False (type: <class 'bool'>)

Loading training data... Loading data from TAB separated file: examples/rsc15/processed/rsc15_train_tr.txt Started training The dataframe is not sorted by SessionId, sorting now Data is sorted in 46.12 Traceback (most recent call last): File "run.py", line 109, in gru.fit(data, sample_store=args.sample_store_size, store_type='gpu') File "/home/../GRU4Rec/gru4rec.py", line 556, in fit generate_samples = theano.function([], updates=updates_st) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/compile/function .py", line 317, in function output_keys=output_keys) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/compile/pfunc.py ", line 486, in pfunc output_keys=output_keys) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/compile/function _module.py", line 1841, in orig_function fn = m.create(defaults) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/compile/function _module.py", line 1715, in create input_storage=input_storage_lists, storage_map=storage_map) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/link.py", li ne 699, in make_thunk storage_map=storage_map)[:3] File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/vm.py", line 1091, in make_all impl=impl)) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/op.py", line 955, in make_thunk no_recycling) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/op.py", line 858, in make_c_thunk output_storage=node_output_storage) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk keep_lock=keep_lock) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in compile keep_lock=keep_lock) File "/home/../anaconda2/envs/gru4rec/lib/python3.6/site-packages/theano/gof/cc.py", line 1641, in cthunk_factory *(in_storage + out_storage + orphd)) RuntimeError: ('The following error happened while compiling the node', GpuBinarySearchSorted{context_name=None, dtype_int64=True}(GpuFromHost.0, GpuFromHost.0), '\n', 'GpuKernel_init error 3: nvrtcCompileProgram: NVRTC_ERROR_BUILTIN_OPERATION_FAILURE')

OS: Debian 4.9.110-3+deb9u4~deb8u1 (2018-08-24) x86_64 GNU/Linux cudnn: 7.6 cuda: 9.2 theano: 1.0.4 pygpu: 0.7.6 libgpuarray: 0.7.6

Anyone could help?

[hidasib]

This is not a GRU4Rec related error, but a sign that something in your Theano setup is not correct (it's not even the fault of Theano, but there is some kind of incompatibility between the driver/cuda/cuDNN on your system).

Some ideas on what could have gone wrong:

• Maybe your nVidia driver is too old (as the error message suggests). You should have at least r396.26, but it also works with newer (for example I use r430.14). If your driver is old, try upgrading it. (When upgrading the driver pay attention to the runfile vs. package issue.)
• Maybe the cuDNN you use was built for a different cuda version (there are different cuDNN 7.6 libs built for cuda 9.2, cuda 10.0, cuda 10.2, etc). Double check that your cuda and cuDNN match.
• Maybe when you built libgpuarray/pygpu, you did it in a different environment and it links to a different cuda/cuDNN version than you are using currently.
• It is less likely, but maybe a different cuda version is in your PATH or in LD_LIBRARY_PATH or in CUDA_HOME (or these environment variables don't point to any cuda version).
• You can also try disabling cuDNN usage in Theano by running it with the appropriate flag: THEANO_FLAGS=dnn.enabled=False python run.py .... I don't think that this has a significant impact on the speed of GRU4Rec, but I haven't tried yet. If setting this flag helps, the issue is definitely some kind of incompatibility between your cuDNN and cuda (and/or nVidia driver).
• You can also try to delete the theano cache (theano-cache purge or manually deleting the contents of ~/.theano/), maybe something is stuck there from a previous version.

If nothing else works, you can set up the whole environment from the ground up. I usually do it this way, because then I know exactly what was installed. It has worked for me 100% of the time. The main steps are:

• Uninstall your current cuda and nVidia driver (if these are installed)
• Install the nVidia driver (>=396.26)
• Install cuda 9.2
• Set the CUDA_HOME environment variable to <cuda_install_dir> (default: /usr/local/cuda-9.2), and make sure that <cuda_install_dir>/bin and <cuda_install_dir>/lib64 are added to PATH and LD_LIBRARY_PATH respectively.
• Download the appropriate cuDNN libs and copy them under cuda/lib64
• (If you start on a new machine, this is the point where you also install python and the required packages (optionally in pyenv/virtualenv))
• Install pycuda
• Build and install libgpuarray & pygpu
• Build and install Theano

[Abigale001]

Thank you very much. I will check the problem according to your comments.