Open cypamigon opened 1 week ago
Hello Cypamigon,
After some investigations with the provided yaml file we couldn't replicate the issue regarding best_weights.h5
not being present in /experiments_outputs/"%Y_%m_%d_%H_%M_%S"/saved_models/
.
Because you are on Windows maybe you forgot to change the 256 characters maximum path length.
To change this you can follow instructions in the TIP
section of the main README
(at the end).
Thanks,
Thanks for your quick feedback.
Unfortunately, I've already enabled windows long path support. I've tried to change the output path but it behave the same.
Ok, another explanation could be that the ssd_mobilenet_v2_fpnlite_035_416.h5
model we provide, trained on person detection
kept the information about its previous training especially the best val_loss
.
And when you try to save the best_weights.h5
it does not save anything because the new val_loss
of your training is higher then the best val_loss
.
If this is true a workaround could be -> for just 1 epoch put save_best_only=False
then stop the training, use the best_weights.h5
of this training (best_weights.h5
in general.model_path
section) to launch another training but with save_best_only=True
this time.
Thanks,
Hmm, okay, looks promising. I'm currently running a training session with save_best_only=False
. I'll try your solution once it finishes.
Thanks!
Hello,
I'm trying to train an object detection model based on a custom dataset. I'm following the instructions provided in the README of the object_detection/src folder.
I've modified the
user_config.yaml
file according to my need and I'm running the training script withpython stm32ai_main.py
.According to the instructions, best model weights since the beginning of the training should be automatically saved on the
/experiments_outputs/"%Y_%m_%d_%H_%M_%S"/saved_models/
folder. However, the weights are never saved during the training (nobest_weights.h5
in the folder).At the end of the training process, when the scripts want to load the weights, an error is raised because the path doesn't exist !
I've tried to modify the keras.callbacks.ModelCheckpoint parameters to saved the weights at the end of each epoch (even if they are not the best) and it works (
best_weights.h5
are saved in the saved_models folder).*I've replace :
with :
However, I would like to save the best weights since the begining of the training in order to get the more efficient model. Do you have any idea on what could prevent the script to save the
best_weights.h5
file whensave_best_only
parameter is set toTrue
?I'm running the script on Windows 10 and in a
st_zoo
virtual env as detailled in the repository README.Here is my
user_config.yaml
file :