Closed daniel-j-h closed 5 years ago
Users can
and then e.g. fine tune starting from that checkpoint.
The problem we are now running into is we load not only the checkpoint but also the optimizer including e.g. the learning rate.
What's currently not possible
since we just ignore the r1 learning rate from the user and load the initial learning rate r1 from the serialized checkpoint into our optimizer.
Check out
https://github.com/mapbox/robosat/blob/a855a8a0095a8eca284a074680b5eb82faa31abc/robosat/tools/train.py#L81-L95
We should re-think this. cc @ocourtin this might effect you, too.
@daniel-j-h
If you want/need to change learning rate, why still using --resume too ? As you can use --checkpoint flag only ?
Oops you are right, not sure why I missed this :see_no_evil:
Users can
and then e.g. fine tune starting from that checkpoint.
The problem we are now running into is we load not only the checkpoint but also the optimizer including e.g. the learning rate.
What's currently not possible
since we just ignore the r1 learning rate from the user and load the initial learning rate r1 from the serialized checkpoint into our optimizer.
Check out
https://github.com/mapbox/robosat/blob/a855a8a0095a8eca284a074680b5eb82faa31abc/robosat/tools/train.py#L81-L95
We should re-think this. cc @ocourtin this might effect you, too.