search

aigc-apps / CogVideoX-Fun

📹 A more flexible CogVideoX that can generate videos at any resolution and creates videos from images.
Apache License 2.0
442 stars 29 forks source link

GPU memory usage increases during training #30

Open wyyfffff opened 1 month ago

wyyfffff commented 1 month ago

Thank you for your excellent work!

When I am training using train_control.py In 2 a800 gpu, I found my GPU memory usage has been increasing, slowly increasing from 65GB to 73GB. My batch size is 4, video resolution is 512x512. Is there any way to solve this problem?

Thank you!

bubbliiiing commented 1 month ago

This may be caused by the training strategy, as videos of different resolutions and lengths are trained together. If you expect the training memory usage to remain relatively constant, it's best to fix the resolution and length.