search

silviazuffi / smalst

MIT License
163 stars 24 forks source link

cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED #6

Open lorentz-wu opened 4 years ago

lorentz-wu commented 4 years ago

my cuda is 10 when i run script/smalst_op_run.sh , i get some error

outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED

how can i solve it?

silviazuffi commented 4 years ago

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10.

On 14. Feb 2020, at 03:36, lorentz-wu notifications@github.com wrote:

my cuda is 10 when i run script/smalst_op_run.sh , i get some error

outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED

how can i solve it?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/silviazuffi/smalst/issues/6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

lorentz-wu commented 4 years ago

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10. … On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

thank you for your help

lorentz-wu commented 4 years ago

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10. … On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi, could you tell me your cuda's version? I will try the cuda with same version

silviazuffi commented 4 years ago

Yes, 9.2

On 18. Feb 2020, at 02:14, lorentz-wu notifications@github.com wrote:

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10. … <x-msg://14/#> On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6 https://github.com/silviazuffi/smalst/issues/6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi, could you tell me your cuda's version? I will try the cuda with same version

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/silviazuffi/smalst/issues/6?email_source=notifications&email_token=ABNYVFOXSAXGVOIZMW7CL4TRDMZALA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMAGLBQ#issuecomment-587228550, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL2CYSPGYL3VWRXNELRDMZALANCNFSM4KU7KARQ.

QingAL commented 4 years ago

The problem is in chainer that is needed for NMR. You can check if NMR can run in cuda 10. … On 14. Feb 2020, at 03:36, lorentz-wu @.***> wrote: my cuda is 10 when i run script/smalst_op_run.sh , i get some error outputs = self.forward(in_data) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 104, in forward y = _matmul(a, b, self.transa, self.transb) File "/home/xxxx/anaconda3/envs/py27/lib/python2.7/site-packages/chainer/functions/math/matmul.py", line 50, in _matmul return xp.matmul(a, b) File "cupy/core/core.pyx", line 3340, in cupy.core.core.matmul File "cupy/core/core.pyx", line 3489, in cupy.core.core.matmul File "cupy/cuda/cublas.pyx", line 559, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 568, in cupy.cuda.cublas.sgemmBatched File "cupy/cuda/cublas.pyx", line 226, in cupy.cuda.cublas.check_status cupy.cuda.cublas.CUBLASError: CUBLAS_STATUS_EXECUTION_FAILED how can i solve it? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABNYVFO7SSWZ7JSL5NS37KDRCX7TRA5CNFSM4KU7KAR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4INOHSEA>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABNYVFL3NKJDJBCMNRZNMO3RCX7TRANCNFSM4KU7KARQ.

dear silviazuffi, could you tell me your cuda's version? I will try the cuda with same version

hi,Did you solve this problem when use cuda9.2? my cuda version is 9.0 also get this error