Thanks for the amazing work! I wonder if I can combine this with Mechanic (https://github.com/optimizedlearning/mechanic) to avoid tuning both the learning rate and the schedule. My naive attempt to combine the two packages resulted in no progress on test data at all (though there was some progress on the training data). I guess this happens because SF optimizers use different weights for training and evaluation. Is there an easy fix for this issue? I am ready to copy both packages as raw Python scripts and patch them where needed for personal use.
Thanks for the amazing work! I wonder if I can combine this with Mechanic (https://github.com/optimizedlearning/mechanic) to avoid tuning both the learning rate and the schedule. My naive attempt to combine the two packages resulted in no progress on test data at all (though there was some progress on the training data). I guess this happens because SF optimizers use different weights for training and evaluation. Is there an easy fix for this issue? I am ready to copy both packages as raw Python scripts and patch them where needed for personal use.