ValueError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512].

thoongee commented 2 years ago

Hi, I'm trying to test validate_on_lfw following Wiki. I'm done align the LFW dataset, and I have a hard time on number 6. Run the test.

My computer is Ubuntu Linux. How can I run the test correctly?

2022-07-11 18:15:04.304549: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error 2022-07-11 18:15:04.304574: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: ubuntu 2022-07-11 18:15:04.304579: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: ubuntu 2022-07-11 18:15:04.304641: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 510.47.3 2022-07-11 18:15:04.304655: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 510.47.3 2022-07-11 18:15:04.304660: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 510.47.3 2022-07-11 18:15:04.304854: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. /home/facenet/src/lfw.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray. return np.array(pairs) WARNING:tensorflow:From /home/facenet/src/facenet.py:114: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2.

tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means tf.py_functions can use accelerators such as GPUs as well as being differentiable using a gradient tape.
tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful.

WARNING:tensorflow:From /home/facenet/src/facenet.py:133: batch_join (from tensorflow.python.training.input) is deprecated and will be removed in a future version. Instructions for updating: Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.interleave(...).batch(batch_size) (or padded_batch(...) if dynamic_pad=True). WARNING:tensorflow:From /home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/input.py:732: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. WARNING:tensorflow:From /home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/input.py:732: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version. Instructions for updating: To construct input pipelines, use the tf.data module. Model directory: /home/models/facenet/20180402-114759 Metagraph file: model-20180402-114759.meta Checkpoint file: model-20180402-114759.ckpt-275 2022-07-11 18:15:06.172482: W tensorflow/core/common_runtime/graph_constructor.cc:1526] Importing a graph with a lower producer version 24 into an existing graph with producer version 1087. Shape inference will have run different parts of the graph with different producer versions. Traceback (most recent call last): File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 500, in _import_graph_def_internal results = c_api.TF_GraphImportGraphDefWithResults( tensorflow.python.framework.errors_impl.InvalidArgumentError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512].

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "src/validate_on_lfw.py", line 167, in main(parse_arguments(sys.argv[1:])) File "src/validate_on_lfw.py", line 75, in main facenet.load_model(args.model, input_map=input_map) File "/home/facenet/src/facenet.py", line 383, in load_model saver = tf.train.import_meta_graph(os.path.join(model_exp, meta_file), input_map=input_map) File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1582, in import_meta_graph return _import_meta_graph_with_return_elements(meta_graph_or_file, File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1603, in _import_meta_graph_with_return_elements meta_graph.import_scoped_meta_graph_with_return_elements( File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/meta_graph.py", line 804, in import_scoped_meta_graph_with_return_elements imported_return_elements = importer.import_graph_def( File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/util/deprecation.py", line 561, in new_func return func(*args, **kwargs) File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 404, in import_graph_def return _import_graph_def_internal( File "/home/heaan-venv/lib/python3.8/site-packages/tensorflow/python/framework/importer.py", line 505, in _import_graph_def_internal raise ValueError(str(e)) ValueError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512].

halfroad commented 2 years ago

For those who undergo this problem. I would suggest following actions:

Add the directive "import tensorflow.compat.v1 as tf" to the corresponding .py files.
Use the arguments to specify the model file and pair.txt with absolute full path, as following,

python3 FaceNet/src/validate_on_lfw.py ../Inventory/Aligned /Users/xxxx/Projects/Inventory/Models/20180402-114759.pb --distance_metric 1 --use_flipped_images --subtract_mean --use_fixed_image_standardization --lfw_pairs /Users/xxxx/Projects//FaceNet/data/pairs.txt

michealChin commented 1 year ago

For those who encounter this problem, please refer to this link https://github.com/openvinotoolkit/openvino/pull/11078

HSSH2110 commented 1 year ago

HELP! I got the same issue yet still can not fix it. I have tried different ways :((

omkar3102 commented 11 months ago

same issue cant fix it :(

davidsandberg / facenet

ValueError: Node 'gradients/InceptionResnetV1/Bottleneck/BatchNorm/cond/FusedBatchNorm_1_grad/FusedBatchNormGrad' has an _output_shapes attribute inconsistent with the GraphDef for output #3: Dimension 0 in both shapes must be equal, but are 0 and 512. Shapes are [0] and [512]. #1227