How to train faster_rcnn with hard-background?

[plantroots]

Hello! Does anyone have any idea how I can force feed the model images with potentially tricky objects so that it learns to avoid these biases? I was thinking of adding some background images in the training process (hand picked) with NO LABELS.

My question basically is: will they be tagged as background, and if so, will they end up in the loss function?

[sovit-123]

Hi @plantroots For background images, you can just create an empty XML file. It will not be discarded and will be used an a background image.

[plantroots]

So basically, if I add an image as a background image, but it doesn't have an empty XML, it will be discarded, and if I add an image as background and it has an empty XML it will be automatically added to the background class?

And my final question is: Do I need to have _ background _ as the index 0 class in CLASSES=[] (config.py)?

[sovit-123]

@plantroots Yes, you will need an empty XML file, or else it will be discarded. I know it's a bit of an inconvenience. When I wrote the dataset pipeline, I did not account for this and it's a bit complicated to change that now.

And yes, you do need __background__ as index 0 in config.py.

[plantroots]

Thank you sir! I really appreciate the prompt reply. I will give it a shot now.

[plantroots]

Just to check, would this classify as good empty XML file? example:

[sovit-123]

Yes, I think this also works. I think the dataset.py does not find the bounding box attribute, then it's a background image. So, the above example should also work.

[plantroots]

I also have a version of the repo from https://debuggercafe.com/ from late November/early December 2022. Tell me if I'm wrong, but this background part was not implemented back then, was it?

[sovit-123]

@plantroots No it was not. You will get the best results by using the code from this repo. There is one more thing. By default, the code in this repo applies mosaic augmentation as per the Ultralytics YOLO standard. You can turn off mosaic augmentation by using --no-mosaic flag in the training command. I recommend that you try both with and without mosaic augmentation.

The reason is that originally these models were no trained on mosaic images. So, smaller models sometimes tend to give bad results when mosaic augmentation is applied (but not always). This depends on the complexity of the dataset.

[plantroots]

Is this graph normal for the tensorboard logging (the lines that come from the origin)?

I am experimenting with --no-mosaic since my dataset is quite small. Thank you for the tip.

[sovit-123]

@plantroots No, this is not normal. I have not checked the tensorboard logging for quite some time now. I think its a bit buggy after a recent update. Please refer to the graphs that are stored on the local disk. I will correct the tensorboard logging in the next few days.

[plantroots]

Going back to the empty xml files: if an image has an empty associated xml file, then that entire image is tagged as background? Lets say if I have a 640 x 640 image, then it will be entirely tagged as background ?

[sovit-123]

Yes.

[plantroots]

Thank you for all the replies. I hope these clarifications help others as well in inserting backgrounds! It really helps reduce the false positives!

[sovit-123]

Glad that it helped. @plantroots In case the issue is solved for now, you may close the issue from your side.

[plantroots]

Adding backgrounds/hard-backgrounds is as easy as adding them into your training set and generating and empty .xml for each background image (with matching names of course) and they will end up in your loss function.