varungodbole
commented
5 months ago Thanks for this question!
If you believe that every model between release cycles doesn't change that much, then you can do something like the iterative refinement that we described in the playbook. Essentially, for each release, you would start with the best hyperparameter point from the previous checkpoint as "given". And then proceed with the process outlined in the playbook to discover a new best checkpoint.
Please feel free to reopen the issue if you have more questions.
Hi, thank you for this great guide. I found all the content in this guide benefit me a lot but still some missing parts might exist. In product enviorment, I found that the finetune checkpoint changed more frequently, since there are release pressure and search space can not be establish at once. This raised the question of incremental tuning - how to choose this round’s model as the next round’s fine-tune starting point, and whether the choice would conflict with the next round’s study. I think there might be more complicated and subtle situations that we need to face when identifying scientific, nuisance, and fixed hyperparameters regarding finetune checkpoint of previous round study. I would be very glad to see discussions around this topic.