search

salesforce / LAVIS

LAVIS - A One-stop Library for Language-Vision Intelligence
BSD 3-Clause "New" or "Revised" License
9.81k stars 962 forks source link

Question regarding peak learning rate and minimum learning rate in pretraining stages of BLIP-2 #243

Open L-Hugh opened 1 year ago

L-Hugh commented 1 year ago

Hello, I'm currently working on replicating BLIP-2 and I have some questions regarding the learning rate settings in the two pretraining stages.

  1. Is the peak learning rate set to 1e-4 in both pretraining stages?
  2. In the paper, the minimum learning rate for the second stage is mentioned as 5e-5, but in the configuration file lavis/projects/blip2/train/pretrain_stage2.yaml, the minimum learning rate is set to 1e-5. Can you please clarify the reason for this discrepancy?
  3. What is the minimum learning rate for the first pretraining stage?

I would appreciate your assistance in understanding these settings. Thank you for your time and effort in developing this great project!

Swetha5 commented 1 year ago

Have you been able to reproduce stage 1 and stage 2 of BLIP-2 ?

LiJunnan1992 commented 1 year ago

Thanks for your questions.

  1. Yes the peak learning rate is 1e-4 for both stages.
  2. The minimum learning rate is 5e-5. Please follow the paper for hyperparameter setting, the config file is shown as an example.
  3. The minimum learning rate for stage-1 is 1e-5.
Swetha5 commented 1 year ago

Thanks @LiJunnan1992 for providing the details !

One more question I have is that in the paper you mention that the model is trained for 250k steps in stage 1 and 80k steps in stage2.

In order to properly configure the learning rate scheduler, we also need the values for the following config parameters which are not discussed in the paper:

max_epoch iters_per_epoch

Thanks

linzhiqiu commented 1 year ago

@Swetha5 Is there any update on max_epoch and iters_per_epoch?