Resume flag in `rs train` loads optimizer state and ignores e.g. learning rate

[daniel-j-h]

Users can

• train a model and get checkpoints, then
• use the resume flag pointing to a checkpoint

and then e.g. fine tune starting from that checkpoint.

The problem we are now running into is we load not only the checkpoint but also the optimizer including e.g. the learning rate.

What's currently not possible

• train for n epochs with learning rate r0
• train for another m epochs with learning rate r1 (where r0 >> r1)

since we just ignore the r1 learning rate from the user and load the initial learning rate r1 from the serialized checkpoint into our optimizer.

Check out

https://github.com/mapbox/robosat/blob/a855a8a0095a8eca284a074680b5eb82faa31abc/robosat/tools/train.py#L81-L95

We should re-think this. cc @ocourtin this might effect you, too.

[ocourtin]

@daniel-j-h

If you want/need to change learning rate, why still using --resume too ? As you can use --checkpoint flag only ?

[daniel-j-h]

Oops you are right, not sure why I missed this :see_no_evil: