search

keras-team / keras-preprocessing

Utilities for working with image data, text data, and sequence data.
Other
1.02k stars 444 forks source link

Added missing parameter to image_data_generator.py #328

Closed Mantas-Skackauskas closed 3 years ago

Mantas-Skackauskas commented 3 years ago

Prevents the apply_brightness_shift() function call to alter the image by min-max scaling it because of the missing scale parameter. Issue link: https://github.com/keras-team/keras-preprocessing/issues/327

Summary (updated Dec 10, 2020)

Problem description

I was using ImageDataGenerator with the parameter 'brightness_range'. This lead to unexpected behaviour where using the brightness range in interval [1,1] altered the image even though nothing should have happened. Looking more into the code, I noticed that the brightness range function generates a value 'brightness' from the uniform distribution with interval specified in the 'brightness_range' parameter, hence, the only value from U(1,1) must be 1. However, when the generated value 'brightness' is passed into the function apply_brightness_shift at line 888 seen below

https://github.com/keras-team/keras-preprocessing/blob/58df11e1145b2088092252c4dba02168c6da2b13/keras_preprocessing/image/image_data_generator.py#L888

the scale parameter is ignored and leads to the function using the default parameter 'scale'=True seen below:

https://github.com/keras-team/keras-preprocessing/blob/58df11e1145b2088092252c4dba02168c6da2b13/keras_preprocessing/image/affine_transformations.py#L215

This leads to a scaled output in range from [0-255] which alters the original image value range image-wise, and therefore performs unintentional min-max scaling to the input. Considering this function should only perform brightness modification and not per image min-max scaling of the values, such unexpected behaviour should not happen.

How I found this issue

I found this issue while using TensorFlow (2.3.1) package with function ImageDataGenerator which still relies on calls to keras API and took some time to locate where the issue is coming from.

How to replicate the problem

from tensorflow.keras.preprocessing.image import ImageDataGenerator
import numpy as np

old_image = np.random.randint(low=20,high=50,size=(1,10,10,1),dtype = 'uint8')

idg = ImageDataGenerator(brightness_range=(1,1)).flow(old_image) # unexpected behaviour

new_image = next(idg)[0].astype('uint8')

# output in range [20,50]
print(f'Original value range {old_image.min()} to {old_image.max()}')
# alters image and min-max scales it to range [0,255]
print(f'Original value range {new_image.min()} to {new_image.max()}')

This leads to the altered image:

Original value range 20 to 49 Original value range 0 to 255

Solution

I propose to set the scale parameter in the line 888 to False which will perform the brightness_range without scaling and will return the expected outcome:

 x = apply_brightness_shift(x, transform_parameters['brightness'], False)

Update (Dec 8, 2020)

After some headache, I found out a number of things:

  1. In order to change brightness of the matrix, linear transformations are performed inside PIL library and that part of the code is actually private. Hence, it is not a simple value multiplication such as "brightness*x".
  2. To solve this issue, we need to leave the functions as they are and add additional line of code to scale our image back to the input value range before returning if scaling is applied.

I suggested this in my updated commit:

  1. I made some examples to show that my solution solves this issue:

For comparison, let's have two functions.

The original apply_brightness_shift(...)

def old_apply_brightness_shift(x, brightness, scale=True):
    if ImageEnhance is None:
        raise ImportError('Using brightness shifts requires PIL. '
                          'Install PIL or Pillow.')
    x = array_to_img(x, scale=scale)
    x = imgenhancer_Brightness = ImageEnhance.Brightness(x)
    x = imgenhancer_Brightness.enhance(brightness)
    x = img_to_array(x)
    return x

The new apply_brightness_shift(...)

def apply_brightness_shift(x, brightness, scale=True):
    if ImageEnhance is None:
        raise ImportError('Using brightness shifts requires PIL. '
                          'Install PIL or Pillow.')
    x_min, x_max = np.min(x), np.max(x)
    local_scale = (x_min < 0) or (x_max > 255)
    x = array_to_img(x, scale=local_scale or scale)
    x = imgenhancer_Brightness = ImageEnhance.Brightness(x)
    x = imgenhancer_Brightness.enhance(brightness)
    x = img_to_array(x)
    if not scale and local_scale:
        x = x / 255 * (x_max - x_min) + x_min
    return x

And let's create a function to compare input versus output and the difference between the two:

def compare_input_output(input_img, brightness, scale, brightness_function):

    vmin=input_img.min()
    vmax=input_img.max()

    output_img = brightness_function(input_img, brightness=brightness, scale=scale)
    diff_img = output_img-input_img

    plt.figure(figsize=(9, 3))

    ax1 = plt.subplot(131)
    ax1.set_title('Input') 
    plt.imshow(input_img, vmin=vmin, vmax=vmax)

    ax2 = plt.subplot(132)
    ax2.set_title('Output') 
    plt.imshow(output_img, vmin=vmin, vmax=vmax)

    ax3 = plt.subplot(133)
    ax3.set_title('Difference') 
    plt.imshow(diff_img, vmin=vmin, vmax=vmax)

    get_stats = lambda img: list(np.array([img.mean(), img.min(), img.max()]).round())
    def print_stats(img, name): 
        print('{0} image mean={1} and range=({2},{3})'.format(name,*get_stats(img)))

    print("Brightness {}, scale {}".format(brightness, scale))
    print_stats(input_img, "Input")
    print_stats(output_img, "Output")
    print_stats(diff_img, "Difference")

Let our image be:

low = -1000
high = 1000
x = np.random.uniform(low=low,high=high,size=(10,10,1))

Then using the compare_input_output we get:

OLD:

old_apply_brightness_shift For scale set to False and brightness set to 1.0, 1.5 and 0.5 of the original brightness:

Brightness 1.0, scale False
Input image mean=-16.0 and range=(-993.0,996.0)
Output image mean=125.0 and range=(0.0,255.0)
Difference image mean=140.0 and range=(-741.0,993.0)

image

Brightness 1.5, scale False
Input image mean=-16.0 and range=(-993.0,996.0)
Output image mean=169.0 and range=(0.0,255.0)
Difference image mean=185.0 and range=(-741.0,993.0)

image

Brightness 0.5, scale False
Input image mean=-16.0 and range=(-993.0,996.0)
Output image mean=62.0 and range=(0.0,127.0)
Difference image mean=78.0 and range=(-869.0,993.0)

image

NEW:

apply_brightness_shift For scale set to False and brightness set to 1.0, 1.5 and 0.5 of the original brightness:

Brightness 1.0, scale False
Input image mean=66.0 and range=(-978.0,983.0)
Output image mean=62.0 and range=(-978.0,983.0)
Difference image mean=-4.0 and range=(-8.0,0.0)

image

Brightness 1.5, scale False
Input image mean=66.0 and range=(-978.0,983.0)
Output image mean=353.0 and range=(-978.0,983.0)
Difference image mean=287.0 and range=(-7.0,638.0)

image

Brightness 0.5, scale False
Input image mean=66.0 and range=(-978.0,983.0)
Output image mean=-460.0 and range=(-978.0,-1.0)
Difference image mean=-526.0 and range=(-985.0,-0.0)

image

Update (Dec 10, 2020)

Added two tests for matrix values outside the expected [0,255] range:

def test_random_brightness_scale_outside_range_positive():
    img = np.ones((1, 1, 3)) * 1024
    zeros = np.zeros((1, 1, 3))
    must_be_1024 = affine_transformations.random_brightness(img, [1, 1], False)
    assert np.array_equal(img, must_be_1024)
    must_be_0 = affine_transformations.random_brightness(img, [1, 1], True)
    assert np.array_equal(zeros, must_be_0)

def test_random_brightness_scale_outside_range_negative():
    img = np.ones((1, 1, 3)) * -1024
    zeros = np.zeros((1, 1, 3))
    must_be_neg_1024 = affine_transformations.random_brightness(img, [1, 1], False)
    assert np.array_equal(img, must_be_neg_1024)
    must_be_0 = affine_transformations.random_brightness(img, [1, 1], True)
    assert np.array_equal(zeros, must_be_0)

Related Issues

https://github.com/keras-team/keras-preprocessing/issues/327

PR Overview

Dref360 commented 3 years ago

Thank you for your PR! This is an important issue.

Using brightness requires a PIL Image which lives in [0, 255]. At this point, we might receive pretty much anything from floats to scaled ints.

While I agree we need to fix this bug, I don't think this is the right fix.

Could we do smart clipping? If it's a float image: scale, apply brightness and clip between 0 and 1 If it's an int image: apply brightness, clip between 0 and 255

Also we might want to investigate if adding Brightness is nothing more than np.clip(img*brightness, 0, 255). If that's the case, then we can do it ourselves I think.

Open to suggestions.

Mantas-Skackauskas commented 3 years ago

Thank you for your PR! This is an important issue.

Using brightness requires a PIL Image which lives in [0, 255]. At this point, we might receive pretty much anything from floats to scaled ints.

While I agree we need to fix this bug, I don't think this is the right fix.

Could we do smart clipping? If it's a float image: scale, apply brightness and clip between 0 and 1 If it's an int image: apply brightness, clip between 0 and 255

Also we might want to investigate if adding Brightness is nothing more than np.clip(img*brightness, 0, 255). If that's the case, then we can do it ourselves I think.

Open to suggestions.

@Dref360 Hello, thank you for the message. I have updated this pull request with a code addition that leaves the code as it is and rescales the values to their original range correctly and supports any input range. I am awaiting your suggestions.

Dref360 commented 3 years ago

Almost ready to merge I think. Could we add some tests to check the cases where we have floats or data not in [0,255]?

Mantas-Skackauskas commented 3 years ago

Almost ready to merge I think. Could we add some tests to check the cases where we have floats or data not in [0,255]?

@Dref360 Thanks, I have added 2 tests for positive and negative values (1024 and -1024 respectively). The tests are similar to the existing test test_random_brightness_scale() right above. Let me know if there is anything else I should add.

marcelbischoff commented 3 years ago

Why use PIL for such a simple operation in the first place?