Lowering TE or Unet average only

konstmish / prodigy

The Prodigy optimizer and its variants for training neural networks.

MIT License

298 stars 17 forks source link

Lowering TE or Unet average only #16

Open trihardseven opened 5 months ago

trihardseven commented 5 months ago

Using Prodigy to train SDXL loras specifically, is it possible to force a smaller learning rate for the TE or UNET only? Something like adjusting the d_coef, but only for one of them.

adefazio commented 5 months ago

Please try out the version in this pull request, it supports different LR values for different layers via setting the layer_scale for each in the param group: https://github.com/konstmish/prodigy/pull/9