search

guoqincode / Open-AnimateAnyone

Unofficial Implementation of Animate Anyone
2.85k stars 230 forks source link

About stage 1 training（out of memory） #102

Closed Mumuwei closed 6 months ago

Mumuwei commented 6 months ago

Thanks for the good work. I had the following problem during training phase 1. I used four 40G A100s. As requested, I set the size of the image to 256 and left the rest of the Settings unchanged. The following error occurred. Thanks.

1431705460400_ pic 1441705460418_ pic 1451705460457_ pic

guoqincode commented 6 months ago

I tried 40G VRAM with res=256 and bs=2.

Mumuwei commented 6 months ago

Unfortunately I tried to change bs to 2, still out of video memory. image

Mumuwei commented 6 months ago

I looked at the dataset.py and found that in stage1 , only the image of UBC dataset was read instead of the sequence. If bs was set to 2, then the input was only 2x3x256x256, why would it still out of memory?

Mumuwei commented 6 months ago

Sorry, it's just another multithreaded program preempting memory, not code related.