tensorflow / tensorflow

An Open Source Machine Learning Framework for Everyone
https://tensorflow.org
Apache License 2.0
186.5k stars 74.31k forks source link

Wrong shape broadcasting during tensor equal check #57751

Open creakseek opened 2 years ago

creakseek commented 2 years ago
Click to expand! ### Issue Type Bug ### Source binary ### Tensorflow Version tfnightly ### Custom Code Yes ### OS Platform and Distribution Linux Ubuntu 20.04 ### Mobile device _No response_ ### Python version 3.9 ### Bazel version _No response_ ### GCC/Compiler version _No response_ ### CUDA/cuDNN version _No response_ ### GPU model and memory _No response_ ### Current Behaviour? when comparing two shape-incompatible tensors (with shape `(3,)` and `(5,)`) using `==` on cpu, tf fails to check shape and silently produce an output. if I switch to CUDA then it has the correct behavior: throws error because the shapes cannot broadcast. ### Standalone code to reproduce the issue ```shell import tensorflow as tf import numpy as np print(tf.__version__) with tf.device("cpu"): tensor = tf.constant([1, 2, 3, 4, 5]) print(tf.experimental.numpy.isreal(tensor) == np.array([True, False, False])) # Pass with tf.device("gpu"): tensor = tf.constant([1, 2, 3, 4, 5]) print(tf.experimental.numpy.isreal(tensor).shape) print(np.array([True, False, False]).shape) print(tf.experimental.numpy.isreal(tensor) == np.array([True, False, False])) # Fail ``` ### Relevant log output ```shell 2.11.0-dev20220919 tf.Tensor(False, shape=(), dtype=bool) (5,) (3,) InvalidArgumentError: {{function_node __wrapped__Equal_device_/job:localhost/replica:0/task:0/device:GPU:0}} required broadcastable shapes [Op:Equal] ```
creakseek commented 2 years ago

Similar issue reproduced in tf.experimental.numpy.isclose:

import tensorflow as tf
import numpy as np

with tf.device('cpu'):
  input_data = tf.constant([[1, 2, 3], [4, 5, 6]])
  def tf_isclose(input_data):
      return tf.experimental.numpy.isclose(
          input_data, np.array([[3, 0], [2, 5]]))
  output = tf_isclose(input_data)
  print(output)

with tf.device('gpu'):
  input_data = tf.constant([[1, 2, 3], [4, 5, 6]])
  def tf_isclose(input_data):
      return tf.experimental.numpy.isclose(
          input_data, np.array([[3, 0], [2, 5]]))
  output = tf_isclose(input_data)
  print(output)

Outputs:

tf.Tensor(False, shape=(), dtype=bool)
InvalidArgumentError: {{function_node __wrapped__Equal_device_/job:localhost/replica:0/task:0/device:GPU:0}} required broadcastable shapes [Op:Equal]
tiruk007 commented 2 years ago

@gadagashwini I was able to reproduce the issue on Colab using TF v2.10. Please find the gist here for reference. Thank you!

cantonios commented 2 years ago

The CPU behavior is actually intentional to match numpy. Using == between tensors returns False if there is a shape mismatch doc. If you use tf.math.equal, it will assert on incompatible dimensions.

Looks like the GPU behavior is the buggy one.

Venkat6871 commented 4 days ago

Hi,

Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.

The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.

Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.