Training InSPyReNet_SwinB.yaml on large dataset

[ds-jpg]

Hi @plemeri

I have a custom dataset of 200K+ images containing real world images of cars , products , humans, animals etc , I wanted to know how many epoch should I train for ? I am using InSPyReNet_SwinB.yaml

I would like to inform that my model still in training and it has reached currently 80 epochs , but I see that the train loss was decreasing gradually for the first 4-5 epochs , then from 5 to 80 epoch the train loss is fluctuating between 0.2 to 0.5 on each iteration .

Also I notice that few images are getting better after a few epochs and then again a few ecochs those images are getting worse. I am unable to see any consistent improvement in training .

Can you please guide me ?

Looking forward to your reply.

[plemeri]

Hello,

I had a few questions about training our model with custom dataset, and they are mostly larger than our method. I'm not sure about the reason why there are some problems there. Maybe I think you need to check if there's any problem with your dataset. Are the annotations are consistent? Are those images' size similar to each other? If so, you need to check the dataset and make sure the size and annotation criteria is consistent. Large dataset often worsen the result if it is not constructed carefully.

I'm currently working on using composite dataset with different image sizes. If you are interested in large dataset, please stay tuned.

Thank you.

[ds-jpg]

Hi @plemeri Thank you for your quick reply. I would like to inform you that our dataset is having same size as the original image and their corresponding masks, Also in your training code I notice that you are already doing a static resize which I think is converting the images to 380x380 so I don't know if the problem is occurring with our dataset ?

Also can you please help me understand what do you mean by checking if annotation criteria is consistent ? and if you can kindly recommend any method to follow while constructing the large dataset ?

Looking forward to hearing back from you.

[plemeri]

Sorry for the late reply. I was trying on different conditions to reproduce such problem, but I couldn't find any trouble with training with large dataset. If you don't mind, could you share the dataset that you've in trouble with? I would like to reproduce the issue and fix it as soon as possible.

[ds-jpg]

Hi @plemeri

Thank you for your reply.

Can you kindly share me your email address so I can share the dataset with you.

Thanks

[plemeri]

taehoon1018@postech.ac.kr

Thanks

[ds-jpg]

Hi @plemeri

I would also like too draw your attention to one more problem which I recently came across , When I inference the inspyrenet_swinb model which is trained on my custom dataset I see that in many images the edges are having background image in it. Can you please tell me why this is occuring?

Thanks and looking forward to hearing back from you.

P.S see attached reference output and input image for a better understanding of the issue.

Problem_compressed.pdf

[plemeri]

This can be easily removed by applying morphological operation called erosion.