Closed go-ahead-maker closed 1 year ago
Yes. Actually, we follow the code in DeiT and do not test model_ema
in our codebase. In the downstream tasks, we only use the model
instead of model_ema
.
In my later experiments, model_ema
also does not work for the current model. It may be more suitable for the lightweight models (FLOPs < 1G), which is a common technique for training those models.
Thanks for your valuable reply!
I want to ask you one more question. If I launch the model_ema
during training, will it affect the training of the original model? Or will EMA copy the original model parameters and update them independently? Because I read the code of EMA, It seems first to copy the original model, and update it after the backward of the original model. So, I suppose that model_ema
dost not affect the original model's weight.
Thank you again for your patience for this basic issue!
Sorry for the late reply.
Yes, model_ema
will not affect the model parameters. It works like ensembling models via weighted parameters.
Thus, it will not affect the original model's performance. But the ema
model usually works better while training, that's why they are often used as teachers in contrastive learning.
In my experiments, though it works better in the middle epochs, it achieves similar result in the end.
So appreciate for your detailed explanations, It do help me a lot. looking forward to your future works~
Hi authors, nice work! I got a basic question about the checkpoint file or pretrained weight For image classification, the saved checkpoint file is a dict like
utils.save_on_master({ 'model': model_without_ddp.state_dict(), 'optimizer': optimizer.state_dict(), 'lr_scheduler': lr_scheduler.state_dict(), 'epoch': epoch, 'model_ema': get_state_dict(model_ema), 'scaler': loss_scaler.state_dict(), 'args': args, 'max_accuracy': max_accuracy, }, checkpoint_path)
. when usingmodel_ema
, the checkpoint dict will contain bothmodel
andmodel_ema
. I wonder which of these two will be used when loading the model into the downstream task. I check theload_checkpoint
function used in UniFormer (follow Swin), and it seems choicemodel
to load. So, will themodel_ema
not be used?