GPU usage can not always 100%

THUDM / CogVideo

Text-to-video generation: CogVideoX (2024) and CogVideo (ICLR 2023)

Apache License 2.0

7.31k stars 672 forks source link

GPU usage can not always 100% #202

Open GFENGG opened 2 weeks ago

GFENGG commented 2 weeks ago

System Info / 系統信息

cuda=12.1 diffusers == 0.30.1

Information / 问题信息

[X] The official example scripts / 官方的示例脚本
[ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

run the finetune script, and the gpu usage is as bellow.

And this issue doesn't seem to be caused by eval, (because i have set the eval interval to 100000). why, and how to fix this problem?

Expected behavior / 期待表现

gpu usage keep 100%

zRzRzRzRzRzRzR commented 2 weeks ago

When fine-tuning, this is normal because fakecp only optimizes the VAE decoder part without touching the VAE encoder part. We haven't paid much attention to this issue before, we will try to check it out

GFENGG commented 2 weeks ago

When fine-tuning, this is normal because fakecp only optimizes the VAE decoder part without touching the VAE encoder part. We haven't paid much attention to this issue before, we will try to check it out

okay, thanks, hope it can be fixed