mvpatel2000
commented
3 months ago EVAL_STANDALONE_END event is missing from the pseudo code definition in the Composer event doc
Ah, this is because the doc focuses on the training loop, and this is only run if you call just trainer.eval(...), which we do not provide docstrings for. We'd love a community PR describing the eval loop events after the train loop if you're up for it!
It looks like the pseudo code is referring to another name for the even, AFTER_EVAL_ALL which does not seem to exist anymore.
Ah, it should be EVAL_AFTER_ALL. I'll fix the typo.
Additional it is not very clear that EVAL_END is called after evaluation on one evaluation dataset, so called several times if there are several datasets.
It should be described in the for loop over evaluators in docstring -- is there a better place you'd like a more helpful description?
Expected behavior
EVAL_STANDALONE_END event is missing from the pseudo code definition in the Composer event doc. It looks like the pseudo code is referring to another name for the even, AFTER_EVAL_ALL which does not seem to exist anymore.
Additional it is not very clear that EVAL_END is called after evaluation on one evaluation dataset, so called several times if there are several datasets.
https://docs.mosaicml.com/projects/composer/en/latest/api_reference/generated/composer.Event.html#composer.Event.EVAL_STANDALONE_END