mAp stop increase after some point.

[yakubilik]

I am trying to train yolact/yolact++ model with a custom dataset. After some point of training mAp is just stucks in a certain space. Until this point, mAp increase normally but when mAp reachs at 30s, it just starts going back and forward. Like 30->28->32->24->29->26....

At first it was stucking between 12-8 then i added a comma to this line 'class_names': ('name'), -> 'class_names': ('name',), and this improved the mAp up to 30s.

Also when i try training with "ImmersiveLimit Cigarette Butt Dataset" mAp starts at 90s. With my custom dataset mAp starts at 0 even though i do transfer learning.

The differences between these datasets my label_map is: " 'label_map': { 3: 1 } " the other one is: " 'label_map': { 0: 1 }"

I have 900 train images and 6000-7000 annotations ImmersiveLimit Cigarette Butt Dataset has 2000 images and 2000 annotations

so, what is wrong with my dataset ?

[AdvancedStyle]

How long did you it take you to get to 90 on the cigarette butts (I'm assuming you are talking about this: https://www.immersivelimit.com/tutorials/train-yolact-with-a-custom-coco-dataset)?

And what hardware?

I'm running a GTX1080 + RTX3070 at full blast and after an hour or so I'm still at only about 12 mAP.

[yakubilik]

Yes that is the dataset i used.
As i remember it was starting around 90's for .50 map and keep increasing.

I ran it on google collab which provides Tesla-K80 gpus.

Can't remember the exact time but it was like 4-5 hours to reach 99 map score.

[AdvancedStyle]

I ran it all night but maxed out at 50...but i have a feeling maybe having it split into 2 GPUs divided the score by 2 or something...because the results were really accurate.

[yakubilik]

Interesting, maybe..

If you'll try training with any of your custom dataset, please let me know the results.

[Elizbellou]

@yakubilik hi there, dont know if you are still interested.. I tried a custom dataset, around 2000 images, 1400 train set with 3 classes (around 10.000 annotations for the train set), yolact.base config, resnet101 backbone and same weights for transfer learning using colab. mAP0.5 reached 65% bbox and 51% mask at about 40-50k iterations. And thats the top with the specific dataset, i tried tweaking learning rate, decay, aspect ratio and scales but nothing really special happened. So, one difference i noticed with yours, is the label_map.. I left it as None cause the id categories in json were in a row , 1, 2, 3.. So, unless its otherwise its better to leave it none, or change the json file. Check this for managing your json files, if you dont know it already: https://github.com/immersive-limit/coco-manager.git I was wondering after how many iterations of yours stopped mAP to increase?

[yakubilik]

Hi, thank you for all the information. I still don't know what is causing this. My label map was different, it was 3,4,5... I even think to change my label files to be in 1,2,3.. order but it wasn't make sense.

I can't remember my iteration count but after like 4,5 hours it stopped.

[walterwgh]

@yakubilik Did you resolve the issue?

i have the same issue with the immersive dataset. starting at 0, but also my mask mAP stays at 0.
Training full night on immersive cigarette butt dataset I do not reach 90 mAP.

Even with my own custom dataset, mask mAP stays at 0. could it be RLE related? some masks are polys, some RLE... ?

Are you on master branch?

thx