Why the parameters of the generator(Stage_C) model need to calculate the gradient when training the control net？

Stability-AI / StableCascade

Official Code for Stable Cascade

MIT License

6.52k stars 530 forks source link

Why the parameters of the generator(Stage_C) model need to calculate the gradient when training the control net？ #112

Open xhh12381 opened 7 months ago

xhh12381 commented 7 months ago

In train_c_controlnet.py, why the parameters of the generator model need to calculate the gradient when training the control net, and only the parameters of the control net are set in the optimizer, I think this will extremely increase the consumption of VRAM, and I would like to know if setting generator.eval().requiresgrad(False) will not affect the control net training.

Unified-Robots commented 7 months ago

In my opinion, the training of the controlnet dependents on the parameters, as the controlnet injects the information into the stageC modules to control the contents of the generated images.