generate synthetic data projecting signs in recorded pictures

[fabolhak]

We could generate synthetic training data with GTRSB or other sign images. The images can be projected in our "environment" to create the synthetically labeled data. Based on the projection parameters we can obtain the bounding boxes easily.

Code lives here: https://github.com/tum-phoenix/drive_ml/tree/master/sign_recognition/utilities/data_augmentation

1st approach

• [x] use GTRSB images and project them into some random camera images we recorded

problems:

• images from GTRSB are only very small (no pictures with big signs possible)
• background from GTRSB and background from our environment can be very different
• not all of our sings are in GTRSB

2nd approach

• [ ] use signs with transparent background
• [ ] project signs into random pictures of recordings (zero class images)
• [ ] evaluate different processing steps to make the projected signs look more realistic
• [ ] use of random parameters (using some kind of normal distribution?)
• [ ] evaluate order of processing steps
• [ ] automate process (e.g. python)
• [ ] validation?

code

• we already have some code

possible processing steps

• Brightness & Contrast in OpenCV
• Artificial Lens Distortion in OpenCV
• Gaussian Noise in OpenCV
• RGB noise in OpenCV
• Motion Blur in OpenCV
• Image Transformation in OpenCV
• shadow
• ...

[fabolhak]

For the Carolo Cup some signs are not in the database, to solve this problem we need other pictures.

1. try to find an approch to generate pictures of this signs
2. Maybe you can cut the boundingbox of the sign and replace it by an other sign picuture from google or other.
3. Deform the sign to put in with a shadow tranformation (constant, linear, polynominal), with changes the RGB values.
4. Mix and use the mean of a set of backgrounnds (eg. two).
5. Evaluate with real camera data

[MaslinuPoimal]

Current status: We have a full synthetic environment generation, so this is superseded by that, however this would enable the generation of more realstic images.

The amount of hand-labeled data available to use is relatively high at this point though.