Dense Layer: Different inference results with different BS

[gombru]

I've seen that I get different results out of a Dense layer during inference with batched inference vs single inference. As in this simple example:

model = tf.keras.layers.Dense(8, activation="relu")
x = tf.ones((128, 8))
x_single = tf.ones((1, 8))
y = model(x)
y_single = model(x_single)

np.testing.assert_almost_equal(y[0,:].numpy(), y_single[0,:].numpy(), 8)

Colab reproducing error.

In some runs the outputs do not match.

• The difference gets substantially amplified when inputs are high precison floats.
• The difference seems to vanish when comparing bigger BS: i.e. BS 16 and BS 128 get the same results.

This is tested with Keras 2.8.0

[tilakrayal]

@gadagashwini, I was able to reproduce the issue in tf v2.8, v2.9 and nightly. Kindly find the gist of it here.

[qlzh727]

Note that the numerical difference threshold TF/Keras usually use is 1e-6, which is larger than the current difference. The difference is caused by lower level ops and hardward. You might see bigger difference if you try to run it on GPU.

I am going to close this issue since this is working as intended, and we don't have anything need to address here.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No