search

facebookresearch / schedule_free

Schedule-Free Optimization in PyTorch
Apache License 2.0
1.78k stars 62 forks source link

How to combine this with Mechanic? #37

Open Yura52 opened 1 month ago

Yura52 commented 1 month ago

Thanks for the amazing work! I wonder if I can combine this with Mechanic (https://github.com/optimizedlearning/mechanic) to avoid tuning both the learning rate and the schedule. My naive attempt to combine the two packages resulted in no progress on test data at all (though there was some progress on the training data). I guess this happens because SF optimizers use different weights for training and evaluation. Is there an easy fix for this issue? I am ready to copy both packages as raw Python scripts and patch them where needed for personal use.

adefazio commented 1 month ago

We haven't had success combining them, as the LR adaptation mechanism seems to under-estimate the learning rate. We are not sure why.

Yura52 commented 1 month ago

I see, thanks for sharing!