search

JDAI-CV / fast-reid

SOTA Re-identification Methods and Toolbox
Apache License 2.0
3.42k stars 837 forks source link

ReID model did not save best checkpoint periodically #665

Closed HaiNguyen2903 closed 2 years ago

HaiNguyen2903 commented 2 years ago

Hi there, I used the ReID model to train on my custom dataset but it seem like the best model was not saved as far as I observed. Here are what I did:

  1. Create a custom dataset in folder datasets as Market1501 format
  2. Define the custom dataset in folder fastreid/data/datasets/custom_data.py
  3. Create config file for training model. Here, I used base of Base-SBS.yml and pretrained market_sbs_R50-ibn model.
  4. Modify model config file. I changed the CHECKPOINT_PERIOD to 10 and EVAL_PERIOD to 1. As I understand, the model will be evaluated on test data every epoch and saved checkpoint every 10 epochs. And the model will be saved as best based on the metric (score_rank_1 + mAP) / 2

But as I observed, during training, there were some epochs that the metric got very high. But when I evaluate saved best model on test dataset, the score is not as the highest metric during training.

It seems like after every period (for example 10 epochs), the best metric is re-init to -1, and it continue to save best model from the next period ? (from epoch 11).

Could you explain this for me please.

Many thanks.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] commented 2 years ago

This issue was closed because it has been inactive for 14 days since being marked as stale.