feat: Added additional events such as on_step_begin, on_optimizer_step, on_substep_end

seshapad commented 3 months ago

Description of the change

Some events like on_step_begin, on_optimizer_step, on_substep_end have been added to the trainer controller callback.

Related issue number

How to verify the PR

By using the events in the trainer controller configuration as trigger. For example:

  controller_metrics:
    - name: trainer_state
      class: TrainingState
  controllers:
    - name: stop_on_training_loss_on_save
      triggers:
        - on_optimizer_step
      rule: trainer_state["epoch"] >= 0.5
      operations:
        - hfcontrols.should_training_stop

Was the PR tested

[ ] I have added >=1 unit test(s) for every new method I have added.
[x] I have ensured all unit tests pass

ashokponkumar commented 3 months ago

@seshapad Does this require a particular transformer version to work? And if so, is it compatible with the version used by fms-hf-tuning?

seshapad commented 3 months ago

@seshapad Does this require a particular transformer version to work? And if so, is it compatible with the version used by fms-hf-tuning?

@ashokponkumar Only on_optimizer_step() has been added in 4.42.0 release. The rest have been available even in older versions and have been around for 2 years.

foundation-model-stack / fms-hf-tuning