Two RandomFlip layers with same seed do not return the same result

[cofri]

System information.

• Have I written custom code: No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04
• TensorFlow installed from (source or binary): pip install
• TensorFlow version (use command below): 2.9.2 / 2.10.1 / 2.11.0
• Python version: 3.8

Describe the problem.

With TensorFlow 2.10+, two RandomFlip layers instantiated with the same seed do not return the same result. For instance, when applying the two random flip operations on an image, I expect that the two augmented images are both flipped or not, with the same output. However, this is not the case: on multiple runs, the two augmented images are not always equal (see standalone code below).

Note that with TensorFlow 2.9.2, the expected behaviour is observed: same result for both RandomFlip layers. Since TensorFlow 2.10, this is not the case anymore.

Note also that for two RandomRotation layers with the same seed, the same rotation is applied on the image, whatever the version of TensorFlow (2.9, 2.10 and 2.11). This seems to be the expected behaviour, compared to the RandomFlip behaviour.

Standalone code to reproduce the issue.

import tensorflow as tf
print(tf.__version__)

x = tf.convert_to_tensor([[[1], [0]]]) # Tensor of shape (1, 2, 1)

# Create two RandomFlip layers with same seed
seed = 42
flip1 = tf.keras.layers.RandomFlip(mode="horizontal", seed=seed)
flip2 = tf.keras.layers.RandomFlip(mode="horizontal", seed=seed)

# Run 100 times random flip with the two layers
# In TensorFlow 2.9, same result every time (100 %)
# In TensorFlow 2.10+, not the same result every time
total = 0
N = 100
for i in range(N):
    y1 = flip1(x)
    y2 = flip2(x)
    if tf.math.reduce_all(tf.equal(y1, y2)):
        total += 1
print(f"y1 / y2 equal? {total / N * 100} %")

[tilakrayal]

@gowthamkpr, I was able to reproduce the issue on tensorflow v2.9, v2.11 and nightly. Kindly find the gist of it here.

[cofri]

Thanks @tilakrayal for the gist. Indeed, the outputs of both RandomFlip layers are 100% identical with v2.9, but not with v2.11 and nightly. It's the same behaviour I observed.

[rajivhockey97]

I found out the issue. I will submit a pr soon.

[rajivhockey97]

I have an issue. I have tensorflow installed through pip. I would want to address the keras issue, change the corresponding file and then test this new version of keras for the changes. When i do pip install it gets installed in the dist-packages for python. How do i change the imported version of tensorflow with these new changes and test my changes. I know the reason for this bug but can't seem to test my changes in keras while importing tensorflow. Can this be completely done over colab instead of local environment? Any guides or tutorials for it would be nice.

In other words while doing development for this bug how do i import tensorflow with the changes i made so that i can test them on a gist like above? I use a virtual env for development and have working knowledge of git. Any help would be greatly appreciated. Tia.

[rajivhockey97]

@sachinprasadhs Would you be able to help me out with my issue

[Simon-Bertrand]

Hello dear,

the same for me with the 2.11.0 version of Tf. This code does not always give the same flip :

seedRandomFlip = np.random.randint(0, 1e8)

self.augment_inputs = tf.keras.Sequential([
    layers.RandomFlip(mode="horizontal", seed=seedRandomFlip),
])

self.augment_labels = tf.keras.Sequential([
    layers.RandomFlip(mode="horizontal", seed=seedRandomFlip),
])

This issue is caused because of the use of np.random.choice([True, False]) in the line 770 and line 772 in image_preprocessing.py which is a function that does not consider the given seed. To repair this bug, you can replace for example np.random.choice([True, False]) by self._random_generator.random_uniform([1]).numpy()[0] > 0.5 which now take in account the given seed. In facts, i guess that we will need two different seed if the mode is horizontal and vertical, because if not, the random boolean wont be different in the both horizontal and vertical flip cases.

This seems to be repaired in the master branch in image_preprocessing.py which is now using the self._random_generator to calculate the random value, maybe we'll need to wait.

[cofri]

Thanks @Simon-Bertrand for your deep analysis. As you said, let's just wait and see. v2.12 is coming soon

[sachinprasadhs]

There has been some changes made in the above mentioned code. Anyways, RandomFlip layer under the hood uses RandomGenerator https://github.com/keras-team/keras/blob/master/keras/engine/base_layer.py#L3802. It takes handles the random number generation using user specified seed value. I'm not sure on what changes the behavior changed since 2.10 version onwards.

[mattdangerw]

@LukeWood could you take a look? Or get this to relevant CV person?

To me it looks like this was a regression introduced with the newer KerasCV changes, but then got reverted here https://github.com/keras-team/keras/commit/af76a1a391bf32ef9fb03615654fd8d5875788c1. So this might just be fixed in 2.12? I def don't have all the context on the latest with image augmentation layers though.

[LukeWood]

@mattdangerw thanks for the heads up, yeah we can take a look.

[cofri]

TF 2.12 seems to fix this bug. However, the bug is still present in the nightly version 2.13.0-dev20230322. See gist here.