search

Tencent / TurboTransformers

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
Other
1.49k stars 198 forks source link

cuda error in Transformer decoder in TurboTransformer V0.3.0 #174

Open aihebaikaishui opened 4 years ago

aihebaikaishui commented 4 years ago

when we use the Transformer decoder in TurboTransformer V0.3.0, a cuda error appears. The error is shown below.

RuntimeError: CUDA error: an illegal memory access was encountered [TT_ERROR] CUDA runtime error: an illegal memory access was encountered /TurboTransformers/turbo_transformers/core/cuda_device_context.cpp:33

We want to apply the TurboTransformer V0.3.0 in Machine Translation, but can not solve this issue. Thanks for any suggestions!

feifeibear commented 4 years ago

Hi, could you update your version to the lastest version v0.4.2? If you still need v0.3.0 support, can you give a commit hash id? I am willing to help you identify the malloc bug in that commit.

aihebaikaishui commented 4 years ago

Hello, thank you for your advice. We have update the version from previous v0.3.0 to lastest version v0.4.2, but the problem still exists. We still face with the CUDA error: [TT_ERROR] CUDA runtime error: an illegal memory access was encountered /workspace/TurboTransformers/turbo_transformers/core/cuda_device_context.cpp:33. We also tried to solve this problem by rebuilding the docker and container, however, it doesn't work. I would appreciate it if you help us to solve this problem. Looking forward to your reply.

feifeibear commented 4 years ago

Hi, I got it. Can you paste your build commands as well as the run scripts which may help me to reproduce the error? I guess you are working on the decoder code, which may be not well-tested.