search

imatif17 / Prototype-Mean-Teacher

16 stars 1 forks source link

Training setting of PMT #3

Closed ygzhu closed 8 months ago

ygzhu commented 8 months ago

Thanks for your contribution. In PMT, there are one BURN_UP_STEP and one PROTOTYPE_WARMUP for the modeling. https://github.com/imatif17/Prototype-Mean-Teacher/blob/e972712a974130bef1459f5782daa7202612e9b4/pmteacher/engine/trainer.py#L317

https://github.com/imatif17/Prototype-Mean-Teacher/blob/e972712a974130bef1459f5782daa7202612e9b4/pmteacher/engine/trainer.py#L331

In the config file , the PROTOTYPE_WARMUP is smaller than BURN_UP_STEP. Is the PROTOTYPE_WARMUP not conducted during training? I wonder the PROTOTYPE_WARMUP is inclued by BURN_UP_STEP or is independent following BURN_UP_STEP?

ygzhu commented 8 months ago

Hello, I have noticed the pretrained model of the PMT. And the parameters are displayed as follow. the parameters

It seems that the BURN_UP_STEP has been conducted in the pretrained stages?

During training with pretrained model, the PROTOTYPE_WARMUP is conducted without the BURN_UP_STEP?

imatif17 commented 8 months ago

Hi, The prototype-warm up is a step just to accumulate some values in the prototype for training stability. It can be performed between warm up and MSDA training. Although, I found it not to be that beneficial in most cases. But, I left it in the code, because it might come handy sometimes. You can try to use it.

imatif17 commented 8 months ago

The iteration is only 64999 because I trained a model upto burn-up step. And when doing the hyper-parameter search, I was starting my training from there instead of scratch(to save time and resource). Now, I think about it I should have set args.resume = True.