Questions for freezing and training process

[BauTobias93]

Hello david8862 and/or community,

I want to fine tune a yolo3_mobilenet_lite pretrained model. I have 3 classes (person, tennis racket and tennis ball) and since these are represented in COCO (with different names), I wanted to use the pretrained weights from COCO. My dataset contains about 40k images with different resolutions but all images have labels.

Q1: Is it better to use ImageNet or COCO in this case? I saw you writing in another issue that weights from the darknet family don´t work well on transfer learning. Do you have any experience with ImageNet and COCO?

Q2: Is it smart to freeze the backbone of a pretrained model with COCO weights or should I use no-freeze? Do you have some experience with that small amount of classes and images?

My training command will look like this: train.py --model_type=yolo3_mobilenet_lite --anchors_path=configs/yolo3_anchors.txt --weights_path=weights/yolo3_mobilenet_lite_416_coco.h5 --annotation_file=custom/annotations_1.txt --classes_path=custom/classes.txt --batch_size=32 --freeze_level=1 --total_epoch=240 --transfer_epoch=20 --init_epoch=40 --eval_online --data_shuffle --eval_epoch_interval=20 --save_eval_checkpoint --learning_rate=0.001

Q3: Do you see an issue with this training command? I am not sure what transfer_epoch refers to, because in the documentation it says that it will use the ImageNet model for transfer learning but I explicitly refer to another weights_path. So I am not sure what it is gonna do. Also, what is good transfer learning epoch count? More than 20?

Q4: In the classes_path I am referring to my 3 classes (person, tennis racket and tennis ball), but should I also include all other classes from the COCO dataset or just my class names for my annotations. Should my annotation indexes be the same as in the pretrained model, or can the be one, two and three?

I am fairly new to this, so I hope I don´t ask too many dumb questions. Thanks already for reading and answering my questions. Best regards, Tobias

[david8862]

@BauTobias93 see inline

Hello david8862 and/or community,

I want to fine tune a yolo3_mobilenet_lite pretrained model. I have 3 classes (person, tennis racket and tennis ball) and since these are represented in COCO (with different names), I wanted to use the pretrained weights from COCO. My dataset contains about 40k images with different resolutions but all images have labels.

Q1: Is it better to use ImageNet or COCO in this case? I saw you writing in another issue that weights from the darknet family don´t work well on transfer learning. Do you have any experience with ImageNet and COCO?

[david8862]For normal transfer training process ImageNet pretrained weights should be used with --freeze_level=1 for some transfer epochs, but you can also try to load COCO weights directly (need to enable skip_mismatch to bypass load error). If so you can bypass the transfer stage and unfreeze all layer from beginning.

Q2: Is it smart to freeze the backbone of a pretrained model with COCO weights or should I use no-freeze? Do you have some experience with that small amount of classes and images?

My training command will look like this: train.py --model_type=yolo3_mobilenet_lite --anchors_path=configs/yolo3_anchors.txt --weights_path=weights/yolo3_mobilenet_lite_416_coco.h5 --annotation_file=custom/annotations_1.txt --classes_path=custom/classes.txt --batch_size=32 --freeze_level=1 --total_epoch=240 --transfer_epoch=20 --init_epoch=40 --eval_online --data_shuffle --eval_epoch_interval=20 --save_eval_checkpoint --learning_rate=0.001

[david8862]As Q1, you can try

train.py --model_type=yolo3_mobilenet_lite
--anchors_path=configs/yolo3_anchors.txt
--weights_path=weights/yolo3_mobilenet_lite_416_coco.h5
--annotation_file=custom/annotations_1.txt
--classes_path=custom/classes.txt
--batch_size=32
--freeze_level=0
--total_epoch=240
--transfer_epoch=0
--init_epoch=0
--eval_online
--data_shuffle
--eval_epoch_interval=20
--save_eval_checkpoint
--learning_rate=0.001

Q3: Do you see an issue with this training command? I am not sure what transfer_epoch refers to, because in the documentation it says that it will use the ImageNet model for transfer learning but I explicitly refer to another weights_path. So I am not sure what it is gonna do. Also, what is good transfer learning epoch count? More than 20?

[david8862]In transfer epoch, layers defined by --freeze_level will be set trainable=False to freeze their weights. Usually 10 or 20 transfer epochs should be enough. In model building code it will load imagenet pretrained weights by default, but if you load your weights with --weights_path it will overwrite imagenet ones.

Q4: In the classes_path I am referring to my 3 classes (person, tennis racket and tennis ball), but should I also include all other classes from the COCO dataset or just my class names for my annotations. Should my annotation indexes be the same as in the pretrained model, or can the be one, two and three?

[david8862]If what you need is a model only support your 3 classes, you should use a class file only containing you class names.

I am fairly new to this, so I hope I don´t ask too many dumb questions. Thanks already for reading and answering my questions. Best regards, Tobias

[BauTobias93]

Thank you very much for your quick answers @david8862 ! I will run the training and report back with my results. I hope it is okay that I will leave this issue open for a few more days so I can ask some follow up questions if needed.

[BauTobias93]

Update: Training went well, I dumped the model, evaluated it with a separated dataset (eval.py) and ran it on single images with yolo.py.

I had some trouble including the dumped model in my own code. Before I used pretrained models from sources like TensorFlow Hub, as I expected to use the dumped model as well. Loading the model with from keras.models import load_model, model = load_model(model_path, compile=False) and predicting with result = model.predict(image). I added your image preprocessing and prediction postprocessing, but my prediction results are wrong anyways (compared using yolo.py with the same model and images).

Can I use a dumped model like that or should I convert it to a pb Model for that purpose? Is there a way around for not generating the inference_model? I am trying to use your script mainly for training and have a less general code for prediction.

Thanks in advance

[david8862]

@BauTobias93 dumped model should be able to run independently as long as it can be load. maybe you can refer validate_yolo.py, which is also a separate inference tool for trained model and support different model format (h5/pb/tflite, etc.)

[mu745511]

@BauTobias93 @david8862 hey can you tell me about the training epochs like after how many epochs you started seeing the convergence and about the class score threshold you set in my case i am trying to use efficientnet b0 imagenet pretrained weights for a single class ( person ) but my class scores seems to be very low in between < 0.2 , and what optimizer you use and did you freeze the backbone or trained without freezing the backbone for some epochs in my case i am not freezing the backbone .

[david8862]

@mu745511 sorry for response delay. Generally you can start from the reference training config cmd:

# python train.py --model_type=yolo3_mobilenet_lite --anchors_path=configs/yolo3_anchors.txt --annotation_file=trainval.txt --classes_path=configs/voc_classes.txt --eval_online --save_eval_checkpoint

in my experience transfer epochs with freezing backbone is usually required. For optimizer --optimizer=adam --learning_rate=0.001 and --optimizer=sgd --learning_rate=0.005 may 2 common options.

[mu745511]

@david8862 thankyou for the response i got some good results on a single class .