KidsWithTokens / MedSegDiff

Medical Image Segmentation with Diffusion Model
MIT License
979 stars 147 forks source link

segmentation_sample with dpm_solver==True drains GPU memory. #102

Closed ToruHironaka closed 1 year ago

ToruHironaka commented 1 year ago

I executed segmentation_sample with dpm_solve==True for BraTS2020 Validation data. The program always drains my GPU memory and crashed with GPU out of memory error. My GPU is V100S with 32GB memory. I have also tried RTX8000 with 48GB memory, but both machine got the GPU out of memory errors. The program segmentation_sample with dpm_solve==False setting has been working and GPU memory kept at 2.5GB constantly till the end of execution. I have a couple questions below.

How much GPU memory does segmentation_sample with dpm_solve==True require? or Should I execute this program with small set of data (e.g. execute with BraTS20_Validation_001, after then execute with BraTS20_Validation_002 for each patient)?

I do not think multi_gpu option working. I assume --multi_gpu argument setting is "0,1,2." Is this correct input argument setting for multi_gpu? or Is there any additional command line arguments or modification of the program I need to do for multi_gpu?

WuJunde commented 1 year ago

it's a known bug caused by pytorch version: https://github.com/WuJunde/MedSegDiff/issues/69, I am fixing it.

PapaMadeleine2022 commented 11 months ago

@ToruHironaka @WuJunde I meet the same question. Have you solved it?

ToruHironaka commented 10 months ago

No, I have not. I think WuJunde is working on.