search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
135.61k stars 27.14k forks source link

Is it possible to add L1/L2 regularization using the trainer class ? #33130

Open mayank64ce opened 3 months ago

mayank64ce commented 3 months ago

Feature request

I want to add L1/L2 regularization to the transformer training.

Motivation

Adding L1/L2 reg can promote sparser models that can accelerate inference and reduce storage.

Your contribution

Not sure.

NielsRogge commented 3 months ago

Hi,

If you want to add L1/L2 regularization to your loss function, you can overwrite the Trainer, as shown here.

mayank64ce commented 3 months ago

Thanks for the reply. Is it possible to pass additional parameters (like regularization weights) to the compute_loss method ?

NielsRogge commented 3 months ago

No, I'm afraid adding those isn't possible, as we already support 114 flags.