duane-edgington
commented
8 months ago resolved by controlling the s3 bucket used as the starting location, making sure only the desired model training weights are in that directory, and not the /training prefix (which will be loaded as part of the initial training process). Eg. https://s3.console.aws.amazon.com/s3/buckets/901103-starting-checkpoint?region=us-west-2&bucketType=general&tab=objects
On my recent run of deepsea-ai train, with parameters --input-s3 s3://901103-models-deploy/megadetector/megafish_ROV_weights.pt --resume True
the log file indicates that instead of using the indicated weights for transfer learning, the model training started with the last run of training of the model. That is not terrible in this case, since the input-s3 .pt file indicated was the original starting point, but it could be unexpected.
@danellecline @duane-edgington