search

JackieHanLab / TOSICA

Transformer for One-Stop Interpretable Cell-type Annotation
MIT License
126 stars 24 forks source link

Memory leakage #16

Open DingMaolinBio opened 1 year ago

DingMaolinBio commented 1 year ago

Hi, I am sorry to bother you. When I tried to apply the TOSICA, I was confused in installing the environment. Here are details:

  1. When I used packages (pytorch=1.7.1 torchvision=0.8.2 torchaudio=0.7.2 cudatoolkit=10.1), the training process would stucked while move the model to the GPU;
  2. When I used the environment that included packages(pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6), the consumption of the GPU would increase while training. I guess it could be the leakage of the memory.

Could you offer us some suggestions about these situation? Thanks for your reply.

GlancerZ commented 1 year ago

Hi, I am sorry to bother you. When I tried to apply the TOSICA, I was confused in installing the environment. Here are details:

  1. When I used packages (pytorch=1.7.1 torchvision=0.8.2 torchaudio=0.7.2 cudatoolkit=10.1), the training process would stucked while move the model to the GPU;
  2. When I used the environment that included packages(pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6), the consumption of the GPU would increase while training. I guess it could be the leakage of the memory.

Could you offer us some suggestions about these situation? Thanks for your reply.

Do you have any idea, I also meet this problem.

DingMaolinBio commented 1 year ago

Thanks for your request. But I am sorry to tell you that I have not make any progress in handling this issue. At last, I tried to train the model with CPU, but it took too much time. I am truly sorry.发自我的 iPhone在 2023年9月23日，12:45，GlancerZ @.***> 写道：

Hi, I am sorry to bother you. When I tried to apply the TOSICA, I was confused in installing the environment. Here are details:

When I used packages (pytorch=1.7.1 torchvision=0.8.2 torchaudio=0.7.2 cudatoolkit=10.1), the training process would stucked while move the model to the GPU; When I used the environment that included packages(pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6), the consumption of the GPU would increase while training. I guess it could be the leakage of the memory.

Could you offer us some suggestions about these situation? Thanks for your reply.

Do you have any idea, I also meet this problem.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

JiaweiChenGo commented 7 months ago

Thank you for your interest in TOSICA. Thanks to @sldyns, we have resolved the issue of continuous GPU memory growth by altering weights = attn to weights = attn.detach() in TOSICA_model.py.