Finetuning

[ktrapeznikov]

• baseline: trained on cam4 standard learning params (default batch size, lr 1e-3, 200 epochs)
• SPCAM training from scratch (function of amount of data) for subsample = 1,2,4,8,16...
  • lr 1e-3
  • epochs 200 ?
  • batch size ?
• SPCAM fine-tuning baseline CAM4 (function of amount data) for subsample = 1,2,4,8,16 ....
  • lr 5e-5
  • epochs 200 ?
  • batch size ?

how do we scale epochs or batch size as a function subsample?

Tasks: (keep all the output vars)

• [ ] Train CAM4,
• [ ] Train SPCAM from scratch SPCAM for subsample = [1,2,4,8, ... 2^10]
• [ ] Train SPCAM from CAM4 for subsample = [1,2,4,8, ... 2^10]

Setup to start:

• scaling S, steps = N/B*T , if N/S -> B/S ?
• try using same lr for fine-tuning and from scratch

Generate plot of skill vs N/S for both strategies