search

trentool / TRENTOOL3

Open-Source MATLAB toolbox for transfer entropy estimation
http://trentool.github.io/TRENTOOL3/
GNU General Public License v3.0
56 stars 26 forks source link

Error using fnearneigh_gpu #19

Open dpdarrow opened 8 years ago

dpdarrow commented 8 years ago

I am working on a huge dataset and having trouble getting the gpu script working. I was able to solve library issues, but I am stuck on this error:

TEprepare.m - line 934: Optimal tau for this dataset is: 0.39 TEprepare.m - line 946: Optimal dimension for this dataset is: 8 TEprepare.m - line 1056: Adding TEprepare structure to original data structure

InteractionDelayReconstruction_calculate.m - line 113:

OPTIMIZING INFORMATION TRANSFER DELAY
  TEfindDelay.m - line 45: Estimating TE for u = 40 ms

Error using fnearneigh_gpu Error detected in the GPU (check console)

Error in TEcallGPUsearch (line 295) [~, distance_p21] = fnearneigh_gpu(single(pointset_p21),single(pointset_p21),k_th,TheilerT,nchunks);

...

The error reported in the console is: invalid device function

I tried recompiling with the latest nvidia cuda, but no luck so far

trentool commented 8 years ago

Hi @dpdarrow , this seems to be a problem with your device, from googling around, I got that you may have to specify a different architecture when compiling the CUDA functions (-arch=compute_xx and -code=sm_xx). Did you try to change these settings when recompiling (you should be able to set them in the CUDA Makefile)?

dpdarrow commented 8 years ago

Thanks. I was able to find improvements from passing the compute flags. I realized that I am getting errors from running make in the cuda directory. Unfortunately, I am getting a reference error:

/usr/bin/g++ -c -DMX_COMPAT_32 -D_GNU_SOURCE -DMATLAB_MEX_FILE -I"/usr/local/MATLAB/R2016a/extern/include" -I"/usr/local/MATLAB/R2016a/simulink/include" -ansi -fexceptions -fPIC -fno-omit-frame-pointer -pthread -std=c++11 -O -DNDEBUG /home/dpdarrow/matlab/TRENTOOL3/cuda/fnearneigh_gpu.cpp -o /tmp/mex_252588407813488_30845/fnearneigh_gpu.o /usr/bin/g++ -pthread -Wl,--no-undefined -shared -O -Wl,--version-script,"/usr/local/MATLAB/R2016a/extern/lib/glnxa64/mexFunction.map" /tmp/mex_252588407813488_30845/fnearneigh_gpu.o -lgpuKnnLibrary -lcudart -lcusparse -lcublas -L. -L/usr/local/cuda/lib64 -Wl,-rpath-link,/usr/local/MATLAB/R2016a/bin/glnxa64 -L"/usr/local/MATLAB/R2016a/bin/glnxa64" -lmx -lmex -lmat -lm -lstdc++ -o fnearneigh_gpu.mexa64 /tmp/mex_252588407813488_30845/fnearneigh_gpu.o: In function mexFunction': fnearneigh_gpu.cpp:(.text+0x249): undefined reference tocudaFindKnn(int, float, float, float, int, int, int, int, int)' collect2: error: ld returned 1 exit status

I was able to adjust the compute code to 52 but this error arises either way. Thank you for your help! I will continue to attempt other solutions.

dpdarrow commented 8 years ago

I saw the other post by @samuelandjw and found his linking fix of removing extern C from gpuKnnLibrary.cu, which seemed to solve the compilation issue. Unfortunately, the gpu run has been seg faulting matlab, so I am hoping to track that down.

dpdarrow commented 6 years ago

I finally determined that the major issue was kernel interruption from the OS on a nondedicated GPU last year. I upgraded to a new computer, and unfortunately, after fixing everything that had been a problem in the past, I am now receiving the same error.

TEfindDelay.m - line 45: Estimating TE for u = 10 msError using fnearneigh_gpu Error detected in the GPU (check console)

Error in TEcallGPUsearch (line 313) [~, distance_p21] = fnearneigh_gpu(single(pointset_p21),single(pointset_p21),k_th,TheilerT,nchunks);

Error in TEsurrogatestats_ensemble (line 880) [ncount] = TEcallGPUsearch(cfg,channelpair,pointsets_concat_1,pointsets_concat_2, ...

Error in TEfindDelay (line 57) TGA_results{uu}=TEsurrogatestats_ensemble(cfgTESS,data);

Error in InteractionDelayReconstruction_calculate (line 115) [dataprep, TEmat] =

TEfindDelay(predicttimevec_u,cfgTESS,dataprep);

console: invalid device function

current cuda library is 8 and the dedicated Gpu is P5000

Any thoughts or help would be greatly appreciated

mwibral commented 6 years ago

Hi,

At least for the Pro cards from nVIDIA you can set them to belong exclusively to the compute process (using nvidia smi) - no display, then, of course (we run our cards in headless servers only).

Michael

On 13.10.2017 18:41, dpdarrow wrote:

I finally determined that the major issue was kernel interruption from the OS on a nondedicated GPU last year. I upgraded to a new computer, and unfortunately, after fixing everything that had been a problem in the past, I am now receiving the same error.

TEfindDelay.m - line 45: Estimating TE for u = 10 msError using
fnearneigh_gpu
Error detected in the GPU (check console)

Error in TEcallGPUsearch (line 313) [~, distance_p21] = fnearneigh_gpu(single(pointset_p21),single(pointset_p21),k_th,TheilerT,nchunks);

Error in TEsurrogatestats_ensemble (line 880) [ncount] = TEcallGPUsearch(cfg,channelpair,pointsets_concat_1,pointsets_concat_2, ...

Error in TEfindDelay (line 57) TGA_results{uu}=TEsurrogatestats_ensemble(cfgTESS,data);

Error in InteractionDelayReconstruction_calculate (line 115) [dataprep, TEmat] =

TEfindDelay(predicttimevec_u,cfgTESS,dataprep);

console: invalid device function

current cuda library is 8 and the dedicated Gpu is P5000

Any thoughts or help would be greatly appreciated

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/trentool/TRENTOOL3/issues/19#issuecomment-336504850, or mute the thread https://github.com/notifications/unsubscribe-auth/AIqYGnzDUl7QI9ekloTeauLWhjHZkC30ks5sr5KggaJpZM4JSKUH.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/trentool/TRENTOOL3","title":"trentool/TRENTOOL3","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/trentool/TRENTOOL3"}},"updates":{"snippets":[{"icon":"PERSON","message":"@dpdarrow in #19: I finally determined that the major issue was kernel interruption from the OS on a nondedicated GPU last year. I upgraded to a new computer, and unfortunately, after fixing everything that had been a problem in the past, I am now receiving the same error. \r\n\r\n\u003e TEfindDelay.m - line 45: Estimating TE for u = 10 msError using fnearneigh_gpu\r\nError detected in the GPU (check console)\r\n\r\nError in TEcallGPUsearch (line 313)\r\n [~, distance_p21] = fnearneigh_gpu(single(pointset_p21),single(pointset_p21),k_th,TheilerT,nchunks);\r\n\r\nError in TEsurrogatestats_ensemble (line 880)\r\n [ncount] = TEcallGPUsearch(cfg,channelpair,pointsets_concat_1,pointsets_concat_2, ...\r\n\r\n\r\nError in TEfindDelay (line 57)\r\n TGA_results{uu}=TEsurrogatestats_ensemble(cfgTESS,data);\r\n\r\nError in InteractionDelayReconstruction_calculate (line 115)\r\n[dataprep, TEmat] = \r\n\r\n\u003e TEfindDelay(predicttimevec_u,cfgTESS,dataprep);\r\n\r\nconsole: invalid device function\r\n\r\n\r\ncurrent cuda library is 8 and the dedicated Gpu is P5000\r\n\r\nAny thoughts or help would be greatly appreciated\r\n"}],"action":{"name":"View Issue","url":"https://github.com/trentool/TRENTOOL3/issues/19#issuecomment-336504850"}}}

--

Prof. Dr. rer. nat. Michael Wibral MEG Labor, Brain Imaging Center Goethe Universität

Heinrich Hoffmann Strasse 10 60528 Frankfurt am Main

Phone: +49 69 6301 83193 Fax: +49 69 6301 83231

Xirailuyo commented 3 years ago

See my answer here: https://github.com/trentool/TRENTOOL3/issues/27