Open Piplebobble opened 3 years ago
HI, as mentioned in the readme, you need to write a program to convert your data set into the form of [x
OK,I have get it ,but I find that my training process is very slow. My input size is 416 * 416,I have 13000 training images and 2500 verification images ,batch size=16,My GPU occupation is as follows, but it still takes 25 minutes to have an epoch,Is this normal?thanks
It’s not normal that an epoch takes 25 minutes (for 13000 images). The original "step_per_epoch=7327" was set for MS-COCO. You can calculate the new number yourself, "step_per_epoch=13000/batch_size", which should speed up a lot
I've changed the value of “step_per_epoch=848” and input size=224*224,and now it takes 5 minutes to train a epoch with gpu 20G occupied, Is this normal?(I don't know if it's slow),thanks!!
I use two 1080ti to train 110,000 images (step_per_epoch=7327). At 224x224, an epoch takes about 30~35 minutes, so it is normal and very fast for an epoch to take five minutes.
After training 50 + 40 + 20 epoch, I evaluated the model,Its map is only 22% (with 13000 training data-VOC). I don't know why,Can you give me some suggestions?( I haven't changed backbone and fpn repeat, anchors or other things, I just changed labels and step per_ epoch and not use the "init_weight_path") and the“fpn_filters=112”,What is it for?thank you!!!
Try changing the focal loss of line 388 in csl_yolo.py to the loss of line 389. I have found that sometimes the training is very unstable:
cnfd_loss=self._ConfidenceFocalLoss(pred_y,true_mask,ignore_mask,true_wht)
replace with
cnfd_loss=self._ConfidenceLoss(pred_y,true_mask,ignore_mask,true_wht)
"fpn_filters" represents the number of channels in all feature maps in CSL-FPN. A larger number usually results in better performance but also slows down the model speed. For details, you can refer to my paper. In addition, you are not the first man to tell me this issue, and I will investigate this issue further in the next few days. thank you too.
hello,you say your data format is different from the official format of MS_COCO,so how can I get JSON files like you? thanks!