Remove shadows from an image as a form of preprocessing

[Oluwatoni]

Given an image of the track remove the shadows in that image

Acceptance Criteria Inputs

• Left/Right camera stream

Outputs

• Image without shadows

Code should be fast enough to run realtime(30Hz)

[TranBrian10]

Implementing the shadow removal process found in: "Shadow Detection and Removal from a Single Image Using LAB Color Space".

[Oluwatoni]

Deadline: 17th of MAy for MATLAB and 24th for Opencv

[TranBrian10]

Test image 1. Works well with default parameters.

THRESHOLD_TOLERANCE = 0.8;
MIN_SHADOW_AREA = 30;
MASK_DILATION = 6;
EXPANDED_SHADOW_MASK_DILATION = 10;
EXPANDED_EDGE_MASK_DILATION = 8;
MED_FILTER_KERNEL_SIZE = 15;

[TranBrian10]

Test image 2. Had to change THRESHOLD_TOLERANCE to 1.5 and MASK_DILATION to 8 to get it to detect shadows. Lines look pretty good before the over-illumination fix, but then get a little messed up because the shadows are pretty spotty (the over-illumination fix applies to edges of shadows). Still, the white lane lines are mostly visible and it's better than detecting no lane lines at all. This is probably the most difficult test image because the majority of the road is covered by spotty shadows.

[TranBrian10]

Test image 3. Had to change THRESHOLD_TOLERANCE to 1. Not much to say here since the shadows don't overlap the lane lines. The "Removed Shadows" stage looks pretty artsy though.

[TranBrian10]

Test image 4. Had to change THRESHOLD_TOLERANCE up to 1.5 since the original image had horribly blown out highlights. Didn't go too well, but this is a very non-ideal image and half the frame is pretty much white anyways.

[TranBrian10]

Test image 5. Works decently well with default parameters. Blurs some of the dotted white lane lines in the distance, but those probably wouldn't be in the region of interest anyways. More importantly, the lane line in the right foreground is brought out of shadow.

[TranBrian10]

Test image 6. Had to change THRESHOLD_TOLERANCE to 1.2 and MASK_DILATION to 8. Great results with the lane line in the foreground, although it cuts a bit off the top of the line with the over-illumination step. I could see this being an issue if a shadow just borders a lane line and the over-illumination fix runs the median filter over the line and erases it.

[TranBrian10]

Test image 7. Works well with default values. Not much to say here - the shadows don't cover the lane lines and the algorithm does its job.

[TranBrian10]

Test image 8. Had to change THRESHOLD_TOLERANCE to 1.2. Shadows don't overlap the lane lines here, and the white and yellow lane lines are mostly intact. However, the over-illumination step wipes out the second-nearest white dotted lane line, which is a bummer, but it's a very small part of the image (not sure if it would be picked up by the lane detection).

[TranBrian10]

Test image 9. Had to change THRESHOLD_TOLERANCE to 1.2. Pretty good results with the dotted white lane line closest to the camera.

[TranBrian10]

Test image 10. Works fine with default parameters. Shadows didn't really overlap lane lines here. The over-illumination median filter didn't go down too well on those trees, but that doesn't really matter.

[TranBrian10]

Test image 11. Works fine with default parameters. Not much going on here.

[TranBrian10]

Feels like the defaults may be a little too strict, but it can't be a single constant because the ideal THRESHOLD_TOLERANCE varies a bit between test images, so it may require some tuning based on average lightness in a photo or something.

Also, the over-illumination fix seems to be a recurring problem, while also being the most time-consuming process of the shadow removal algorithm. But, without it, the edges of the shadow regions are very bright and will be picked up by the white lane detection algorithm. Maybe there's an alternative?

Next step is to implement in OpenCV.

[TranBrian10]

Before:

After:

Test image 1. Recovers some of the white lane line.

Note: the RGB image is the original in both the before and after plots.

[TranBrian10]

Before:

After:

Test image 2. This image had spotty shadows, but the shadow removal was able to bring out some of the lane lines closer to the camera. It recovers some of the right white lane line, which was not previously detected.

[TranBrian10]

Before:

After:

Test image 4. This image had the blown out highlights and thresholding is a mess both before and after.

[TranBrian10]

Before:

After:

Test image 5. Before and after look pretty similar, a result of the threshold being a little too strict. Looking at the value plot, the lane line to the right becomes brighter after the shadow removal.

[TranBrian10]

Before:

After:

Test image 6. One of the better results. The center lane line is recovered.

[TranBrian10]

Before:

After:

Test image 9. This image had highlights on the road, but the thresholding is a little too strict here as well. From the value plot, the dotted white lane line closest to the camera is brightened considerably.

[TranBrian10]

The results indicate that we may have trouble dealing with blown out highlights in the images and it's worth trying to fine tune the threshold value to be more lenient.

[jaku-jaku]

1. Can we just fixing the bottom half (or ground region) of the image. since the closest regions to the robot are the regions matter for u. OR,
2. we can try HDR afterward, to combine two images, so it will fix the over-exposed region.

[TranBrian10]

1. After the extraction of lane lines, I apply a region of interest. I believe I need the full image for at least the shadow removal step because it works based on the average lightness of the image.
2. I haven't seen much footage from the cameras and I don't know if they support HDR, but hopefully they don't have this exposure problem.