Feature request: Stop when loss reaches X

[tensiondriven]

Several people on Reddit have been mentioning ideal loss ranges for training. Some training tools have an option to "stop training when loss reaches X". I would love to use this feature with alpaca_lora_4bit, so I wouldn't have to guess at the ideal loss, save a lot of checkpoints, etc.

Would this be something feasible to implement? I may be missing something about the details which would prevent it from being feasible.

For example, I would love to be able to specify:

  --stop-at-loss 1.5

[johnsmith0031]

The finetune scripts use transformers.Trainer, so you can just utilize the feature of it. You can adjust the finetune.py:

class MyCallback(transformers.TrainerCallback):
        "A callback that prints a message at the beginning of training"

        def on_step_end(self, args, state, control, **kwargs):
            if len(state.log_history) > 0:
                if state.log_history[-1]['loss'] < 1.5:
                    control.should_training_stop = True

...

trainer = transformers.Trainer(
        ...
        callbacks=[MyCallback],
    )

And you can also check its document: https://huggingface.co/docs/transformers/main_classes/callback