tensorflow / models

Models and examples built with TensorFlow
Other
77.05k stars 45.77k forks source link

Overfitting model and EarlyStopping #10741

Open Hikozume opened 2 years ago

Hikozume commented 2 years ago

Prerequisites

Please answer the following question for yourself before submitting an issue.

1. The entire URL of the file you are using

https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md

2. Describe the feature you request

When training the model SSD MobileNet V2 FPNLite 320x320 for 50,000 steps, the model overfitting. It was decided to save ~200 checkpoints and find out which one would give the best results. But checking each takes a huge amount of time and seems not quite the right solution. While trying to find a solution to this problem, I thought about the use case of early stopping. The search for the implementation of Early Stopping did not yield any results.

I would like to know possible ways to solve this problem. Because the current approach seems completely wrong.

3. Additional context

Add any other context about the feature request here.

4. Are you willing to contribute it? (Yes or No)

Yes

chunduriv commented 2 years ago

@Hikozume,

Could you please take a look at this answer and let us know if it helps you?

Duplicate https://github.com/tensorflow/models/issues/9766

Hikozume commented 2 years ago

Yes, it looks like a solution. But there is one point, I conduct training through Docker. The launch is done like this: python model_main_tf2.py --model_dir=models/my_ssd_resnet50_v1_fpn --pipeline_config_path=models/my_ssd_resnet50_v1_fpn/pipeline.config More details: https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html#configure-the-training-pipeline And I can't figure out where to put this code. To which file, etc.