search

allenai / rslearn

A tool for developing remote sensing datasets and models.
Apache License 2.0
5 stars 0 forks source link

Automatically handle top_k checkpoints #67

Closed favyen2 closed 3 weeks ago

favyen2 commented 3 weeks ago

For some projects, we have save_top_k to save the best checkpoints, but when we resume during test/predict we only use last.cpkt.

We should have some kind of option to load the best checkpoint during test/predict? It is okay if it only works with save_top_k=1 (so it restores an arbitrary non-last.ckpt checkpoint present in the checkpoints directory).

Currently I sometimes manually copy the top_k checkpoint to overwrite last.ckpt after training is done so it gets loaded during test/predict...

favyen2 commented 3 weeks ago

The relevant code is actually in rslearn_projects and this issue is resolved in https://github.com/allenai/rslearn_projects/pull/27/commits/740a0413842f0acea4afccfd353cc0bc0c687bac