Open Zyj061 opened 2 years ago
Could you provide a little more information on the way you are running the script? I haven't seen this error before, so it is likely a problem with one of the dependencies (tensorflow most likely). Information that would be helpful for me to assist:
Thank you for your reply. Please see below for information about the script we run:
mlde
, which is created by your tutorial using mlde.yml.python generate_encoding.py transformer GB1_T2Q --fasta ./code/validation/basic_test_data/2GI9.fasta --positions V39 D40 --batches 1
Ok, so it looks like the error begins while calling run_embed.py
from TAPE. The error that you're seeing is well documented on the tensorflow GitHub (Linke here). The most recommended solutions seem to be:
nvidia-smi
.I always find that 1 is a good strategy whenever something strange is going on with CUDA, pytorch, or tensorflow. I do not believe that 2 and 3 should be a problem in MLDE and TAPE (otherwise many other people would have run into this problem already), but adding code to allow memory growth is always an option to try.
Some other tests that you can run to try and narrow down where the problem is coming from:
tape-embed
outside of MLDE to see if you get the same error. Instructions for TAPE and the tape-embed
command are here. You should run this in the mlde
environment. mlde2
environment and runpython generate_encoding.py esm1_t6_43M_UR50S GB1_T2Q --fasta ./code/validation/basic_test_data/2GI9.fasta --positions V39 D40
When test the scirtp `generate_encoding.py', it always interrupt with the error: Blas GEMM launch failed: a.shape=(56, 512), b.shape](512,512), m=56, k=512. We test the code with GPU 3090 that has 24G free memory, so I think it is not due to the out of memory.