Closed sam-writer closed 3 years ago
You can't have different evaluation and save intervals when using load_best_model_at_end=True
(save need to be synchronized with evaluation otherwise we can't keep track of the best model). Remove that option and you will have the evaluation and save disconnected as requested.
Thank you, that makes sense.
Also, now that I know it's related I immediately noticed
Might be worth mentioning under save_strategy
as well? But maybe it was just me.
Sure! Do you want to make a PR with that change?
sure!
haha it's been a while!
Oh indeed! :sweat_smile:
Environment info
transformers
version: 4.6.1Who can help
@sgugger
Information
Model I am using (Bert, XLNet ...): Bert, but I don't think that is relevant
The problem arises when using:
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
TrainingArgs
object witheval_steps < save_steps
andeval_strategy
andsave_strategy
both set to"steps"
Trainer
eval_steps
steps, not everysave_steps
stepsHere is my
TrainingArguments
code:with the values I am using filled in, this is:
Expected behavior
Well, maybe this is expected? But if so, I feel like it should be documented more obviously.
I wrote a callback to upload the saved checkpoint to GCS, but the eval step is very quick, so I was going to do those much more frequently. However, if evaluating means I have to upload to GCS, then I will evaluate less often. However, I verified that even if I don't use the GCS save callback, with the above settings, a checkpoint is saved every 2 steps, not every 6.
If this is expected behavior, then is the correct way to change it to write a Callback that
on_evaluate
sets the argument of typetransformers.TrainerControl
to have propertyshould_save
toFalse
?Thank you