Open sandeep1404 opened 10 months ago
Hi @sandeep1404 In TF 2.x, eager execution is enabled by default and you are using AIMET TensorFlow CLE API designed for Graph mode (uses tf.Session/tf.Graph
). You will need to disable eager execution in order to use it. To disable eager execution, please put tf.compat.v1.disable_eager_excution()
API before instantiating the model.
import tensorflow as tf
tf.compat.v1.disable_eager_execution()
Also, if you are interested in using TF eager execution mode, you can also take a look at AIMET Keras CLE API as well which is designed for eager execution.
https://quic.github.io/aimet-pages/releases/latest/api_docs/keras_cross_layer_equalization.html#api-keras-cle
@quic-hitameht , Thanks for the reply but I already disabled eager execution and run the code, after disabling eager execution only i am getting the above error. Can you please check that once. Thank you in advance.
@sandeep1404 Could you please share your complete script to reproduce the issue? Thank you.
Hi @quic-hitameht , Thanks for the response, PFB my code the code is running fine but i cannot able to understand whats going inside the CLE, I tried to interpret the logs but i couldnt get information from it, I tried visualizing the new session graph in tensorboard, but the graph_cle
folder is being created for the new session and when i open tensorboard with the logdir as graph_cle
using the command tensorboard --logdir=graphs_cle
it gives the following logs and when i open the localhost link http://localhost:6006/
it says This site can’t be reached
2023-11-03 05:47:16.784452: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-03 05:47:16.851571: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-03 05:47:16.868884: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-03 05:47:17.148442: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-11-03 05:47:17.148500: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-11-03 05:47:17.148503: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-11-03 05:47:17.526458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 05:47:17.531403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 05:47:17.531544: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
W1103 05:47:17.557937 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.559506 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.565727 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.568787 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.570783 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.572910 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.574856 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.575885 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.577856 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.578965 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.580727 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.
W1103 05:47:17.582290 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.10.1 at http://localhost:6006/ (Press CTRL+C to quit)
can you please kindly check the code and report whats the issue and how to understand whats going inside. Thankyou in advance.
Here is my code
# tensorflow
import tensorflow as tf
tf.compat.v1.disable_eager_execution()
from tensorflow.keras.applications.resnet50 import ResNet50
# Cross layer Equalization related imports
from aimet_tensorflow.cross_layer_equalization import equalize_model
# def cross_layer_equalization_auto():
""" perform auto cross layer equalization """
# load a model
tf.compat.v1.keras.backend.clear_session()
model_resnet = ResNet50(weights='imagenet', input_shape=(224, 224, 3))
sess = tf.compat.v1.keras.backend.get_session()
# op=sess.graph.get_operations()
# print(op[-1])
# get starting op name to invoke api for cle
input_op_name = 'input_1'
output_op_name = 'init'
# Equalize a model with Batchnorms
# Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
# use the new session returned for further evaluations on TF graph
with sess.as_default():
new_session = equalize_model(sess, input_op_name, output_op_name)
tf_tensorboard_writer = tf.compat.v1.summary.FileWriter('graphs_cle', new_session.graph)
# op_new=new_session.graph.get_operations()
# print(op_new[2])
# [print(m.values()) for m in op_new]
tf_tensorboard_writer.close()
sess.close()
Here are the logs of the outputs:
2023-11-03 04:56:41.819730: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-03 04:56:41.821519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:41.821685: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:41.821770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.178780: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179428: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
2023-11-03 04:56:42.268941: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
WARNING:tensorflow:From /tmp/ipykernel_3416/1158554977.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 04:56:43,737 - tensorflow - WARNING - From /tmp/ipykernel_3416/1158554977.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 04:56:44.702115: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702340: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702475: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-03_04:56:43.974220/temp
2023-11-03 04:56:45,042 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-03_04:56:43.974220/temp
2023-11-03 04:56:46.537949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538108: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538171: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538292: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538357: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./replace_relu6_with_relu_2023-11-03_04:56:45.693136/temp
2023-11-03 04:56:46,867 - tensorflow - INFO - Restoring parameters from ./replace_relu6_with_relu_2023-11-03_04:56:45.693136/temp
2023-11-03 04:56:48.083233: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083843: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083919: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_cls_2023-11-03_04:56:47.416744/temp
2023-11-03 04:56:48,633 - tensorflow - INFO - Restoring parameters from ./temp_cls_2023-11-03_04:56:47.416744/temp
2023-11-03 04:56:48,984 - CrossLayerEqualization - ERROR - High Bias folding is not supported for models without BatchNorm Layers
@sandeep1404 Thanks for sharing your script. equalize_model
API expects list of input and output names of given model. In your script, you have provided incorrect output_op_name
. For tf.keras.applications.ResNet50()
, it should be predictions/Softmax
.
Regarding your question to understand CLE, the feature attempts to equalize weight ranges across different channels. It is beneficial for models where weight ranges for different channels in a layer has wide variance. You can use the returned session for further evaluations on TF graph.
Hi @quic-hitameht , I changed the output_op_name
to predictions/Softmax
but when i run the code it has some key error, here are the logs when i change the output name:
WARNING:tensorflow:From /tmp/ipykernel_8450/2048079406.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 09:00:06,897 - tensorflow - WARNING - From /tmp/ipykernel_8450/2048079406.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
2023-11-03 09:00:07,116 - Utils - INFO - ...... subset to store [conv1_conv/Conv2D, conv1_bn]
2023-11-03 09:00:07,116 - Utils - INFO - ...... subset to store [conv1_conv/Conv2D, conv1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_1_conv/Conv2D, conv2_block1_1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_1_conv/Conv2D, conv2_block1_1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_2_conv/Conv2D, conv2_block1_2_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_2_conv/Conv2D, conv2_block1_2_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv2_block1_3_conv/Conv2D, conv2_block1_3_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv2_block1_3_conv/Conv2D, conv2_block1_3_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv3_block1_1_conv/Conv2D, conv3_block1_1_bn]
2023-11-03 09:00:07,119 - Utils - INFO - ...... subset to store [conv3_block1_1_conv/Conv2D, conv3_block1_1_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_2_conv/Conv2D, conv3_block1_2_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_2_conv/Conv2D, conv3_block1_2_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_3_conv/Conv2D, conv3_block1_3_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv3_block1_3_conv/Conv2D, conv3_block1_3_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_1_conv/Conv2D, conv4_block1_1_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_1_conv/Conv2D, conv4_block1_1_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_2_conv/Conv2D, conv4_block1_2_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_2_conv/Conv2D, conv4_block1_2_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_3_conv/Conv2D, conv4_block1_3_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_3_conv/Conv2D, conv4_block1_3_bn]
2023-11-03 09:00:07,123 - Utils - INFO - ...... subset to store [conv5_block1_1_conv/Conv2D, conv5_block1_1_bn]
2023-11-03 09:00:07,123 - Utils - INFO - ...... subset to store [conv5_block1_1_conv/Conv2D, conv5_block1_1_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_2_conv/Conv2D, conv5_block1_2_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_2_conv/Conv2D, conv5_block1_2_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_3_conv/Conv2D, conv5_block1_3_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block1_3_conv/Conv2D, conv5_block1_3_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block3_1_conv/Conv2D, conv5_block3_1_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block3_1_conv/Conv2D, conv5_block3_1_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_2_conv/Conv2D, conv5_block3_2_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_2_conv/Conv2D, conv5_block3_2_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_3_conv/Conv2D, conv5_block3_3_bn]
2023-11-03 09:00:07,127 - Utils - INFO - ...... subset to store [conv5_block2_1_conv/Conv2D, conv5_block2_1_bn]
2023-11-03 09:00:07,127 - Utils - INFO - ...... subset to store [conv5_block2_1_conv/Conv2D, conv5_block2_1_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_2_conv/Conv2D, conv5_block2_2_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_2_conv/Conv2D, conv5_block2_2_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_3_conv/Conv2D, conv5_block2_3_bn]
2023-11-03 09:00:07,129 - Utils - INFO - ...... subset to store [conv5_block1_0_conv/Conv2D, conv5_block1_0_bn]
2023-11-03 09:00:07,129 - Utils - INFO - ...... subset to store [conv4_block6_1_conv/Conv2D, conv4_block6_1_bn]
2023-11-03 09:00:07,130 - Utils - INFO - ...... subset to store [conv4_block6_1_conv/Conv2D, conv4_block6_1_bn]
2023-11-03 09:00:07,133 - Utils - INFO - ...... subset to store [conv4_block6_2_conv/Conv2D, conv4_block6_2_bn]
2023-11-03 09:00:07,133 - Utils - INFO - ...... subset to store [conv4_block6_2_conv/Conv2D, conv4_block6_2_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block6_3_conv/Conv2D, conv4_block6_3_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block5_1_conv/Conv2D, conv4_block5_1_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block5_1_conv/Conv2D, conv4_block5_1_bn]
2023-11-03 09:00:07,135 - Utils - INFO - ...... subset to store [conv4_block5_2_conv/Conv2D, conv4_block5_2_bn]
2023-11-03 09:00:07,135 - Utils - INFO - ...... subset to store [conv4_block5_2_conv/Conv2D, conv4_block5_2_bn]
2023-11-03 09:00:07,136 - Utils - INFO - ...... subset to store [conv4_block5_3_conv/Conv2D, conv4_block5_3_bn]
2023-11-03 09:00:07,136 - Utils - INFO - ...... subset to store [conv4_block4_1_conv/Conv2D, conv4_block4_1_bn]
2023-11-03 09:00:07,137 - Utils - INFO - ...... subset to store [conv4_block4_1_conv/Conv2D, conv4_block4_1_bn]
2023-11-03 09:00:07,137 - Utils - INFO - ...... subset to store [conv4_block4_2_conv/Conv2D, conv4_block4_2_bn]
2023-11-03 09:00:07,138 - Utils - INFO - ...... subset to store [conv4_block4_2_conv/Conv2D, conv4_block4_2_bn]
2023-11-03 09:00:07,138 - Utils - INFO - ...... subset to store [conv4_block4_3_conv/Conv2D, conv4_block4_3_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_1_conv/Conv2D, conv4_block3_1_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_1_conv/Conv2D, conv4_block3_1_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_2_conv/Conv2D, conv4_block3_2_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block3_2_conv/Conv2D, conv4_block3_2_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block3_3_conv/Conv2D, conv4_block3_3_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block2_1_conv/Conv2D, conv4_block2_1_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_1_conv/Conv2D, conv4_block2_1_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_2_conv/Conv2D, conv4_block2_2_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_2_conv/Conv2D, conv4_block2_2_bn]
2023-11-03 09:00:07,142 - Utils - INFO - ...... subset to store [conv4_block2_3_conv/Conv2D, conv4_block2_3_bn]
2023-11-03 09:00:07,143 - Utils - INFO - ...... subset to store [conv4_block1_0_conv/Conv2D, conv4_block1_0_bn]
2023-11-03 09:00:07,143 - Utils - INFO - ...... subset to store [conv3_block4_1_conv/Conv2D, conv3_block4_1_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_1_conv/Conv2D, conv3_block4_1_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_2_conv/Conv2D, conv3_block4_2_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_2_conv/Conv2D, conv3_block4_2_bn]
2023-11-03 09:00:07,145 - Utils - INFO - ...... subset to store [conv3_block4_3_conv/Conv2D, conv3_block4_3_bn]
2023-11-03 09:00:07,146 - Utils - INFO - ...... subset to store [conv3_block3_1_conv/Conv2D, conv3_block3_1_bn]
2023-11-03 09:00:07,146 - Utils - INFO - ...... subset to store [conv3_block3_1_conv/Conv2D, conv3_block3_1_bn]
2023-11-03 09:00:07,147 - Utils - INFO - ...... subset to store [conv3_block3_2_conv/Conv2D, conv3_block3_2_bn]
2023-11-03 09:00:07,147 - Utils - INFO - ...... subset to store [conv3_block3_2_conv/Conv2D, conv3_block3_2_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block3_3_conv/Conv2D, conv3_block3_3_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block2_1_conv/Conv2D, conv3_block2_1_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block2_1_conv/Conv2D, conv3_block2_1_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_2_conv/Conv2D, conv3_block2_2_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_2_conv/Conv2D, conv3_block2_2_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_3_conv/Conv2D, conv3_block2_3_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block1_0_conv/Conv2D, conv3_block1_0_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_1_conv/Conv2D, conv2_block3_1_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_1_conv/Conv2D, conv2_block3_1_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_2_conv/Conv2D, conv2_block3_2_bn]
2023-11-03 09:00:07,151 - Utils - INFO - ...... subset to store [conv2_block3_2_conv/Conv2D, conv2_block3_2_bn]
2023-11-03 09:00:07,151 - Utils - INFO - ...... subset to store [conv2_block3_3_conv/Conv2D, conv2_block3_3_bn]
2023-11-03 09:00:07,152 - Utils - INFO - ...... subset to store [conv2_block2_1_conv/Conv2D, conv2_block2_1_bn]
2023-11-03 09:00:07,152 - Utils - INFO - ...... subset to store [conv2_block2_1_conv/Conv2D, conv2_block2_1_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_2_conv/Conv2D, conv2_block2_2_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_2_conv/Conv2D, conv2_block2_2_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_3_conv/Conv2D, conv2_block2_3_bn]
2023-11-03 09:00:07,154 - Utils - INFO - ...... subset to store [conv2_block1_0_conv/Conv2D, conv2_block1_0_bn]
2023-11-03 09:00:07.588003: W tensorflow/c/c_api.cc:349] Operation '{name:'conv1_relu/Relu' id:77 op device:{requested: '', assigned: ''} def:{{{node conv1_relu/Relu}} = Relu[T=DT_FLOAT](conv1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:08.147625: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_1_relu/Relu' id:221 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:08.936135: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_2_relu/Relu' id:292 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:09.711713: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_add/add' id:363 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_0_bn/cond/Identity, conv2_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:10.443266: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_add/add' id:363 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_0_conv/BiasAdd:0, conv2_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:11.163123: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_1_relu/Relu' id:435 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:11.954976: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_2_relu/Relu' id:506 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:12.688118: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_add/add' id:577 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_out/Relu, conv2_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:13.299646: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_1_relu/Relu' id:649 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:13.896197: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_2_relu/Relu' id:720 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:14.492094: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_add/add' id:791 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_add/add}} = AddV2[T=DT_FLOAT](conv2_block2_out/Relu, conv2_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:15.089717: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_1_relu/Relu' id:933 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:15.713069: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_2_relu/Relu' id:1004 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:16.363501: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_add/add' id:1075 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_0_bn/cond/Identity, conv3_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:17.035267: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_add/add' id:1075 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_0_conv/BiasAdd:0, conv3_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:17.673926: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_1_relu/Relu' id:1147 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:18.419134: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_2_relu/Relu' id:1218 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:19.247085: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_add/add' id:1289 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_out/Relu, conv3_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:20.006269: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_1_relu/Relu' id:1361 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:20.701548: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_2_relu/Relu' id:1432 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:21.338198: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_add/add' id:1503 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_add/add}} = AddV2[T=DT_FLOAT](conv3_block2_out/Relu, conv3_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:22.048276: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_1_relu/Relu' id:1575 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block4_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:22.755388: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_2_relu/Relu' id:1646 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block4_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:23.630489: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_add/add' id:1717 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_add/add}} = AddV2[T=DT_FLOAT](conv3_block3_out/Relu, conv3_block4_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:24.498999: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_1_relu/Relu' id:1869 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:25.486446: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_2_relu/Relu' id:1940 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:26.267102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_add/add' id:2021 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_0_bn/cond/Identity, conv4_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:27.074924: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_add/add' id:2021 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_0_conv/BiasAdd:0, conv4_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:27.814852: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_1_relu/Relu' id:2093 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:28.713874: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_2_relu/Relu' id:2164 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:29.490330: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_add/add' id:2245 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_out/Relu, conv4_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:30.251893: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_1_relu/Relu' id:2317 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:31.101687: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_2_relu/Relu' id:2388 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:31.850806: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_add/add' id:2469 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_add/add}} = AddV2[T=DT_FLOAT](conv4_block2_out/Relu, conv4_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:32.613102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_1_relu/Relu' id:2541 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block4_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:33.422959: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_2_relu/Relu' id:2612 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block4_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:34.179514: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_add/add' id:2693 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_add/add}} = AddV2[T=DT_FLOAT](conv4_block3_out/Relu, conv4_block4_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:34.946318: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_1_relu/Relu' id:2765 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block5_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:35.775102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_2_relu/Relu' id:2836 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block5_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:36.532354: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_add/add' id:2917 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_add/add}} = AddV2[T=DT_FLOAT](conv4_block4_out/Relu, conv4_block5_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:37.313127: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_1_relu/Relu' id:2989 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block6_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:38.166129: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_2_relu/Relu' id:3060 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block6_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:38.963474: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_add/add' id:3141 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_add/add}} = AddV2[T=DT_FLOAT](conv4_block5_out/Relu, conv4_block6_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:39.889524: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_1_relu/Relu' id:3293 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:41.284167: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_2_relu/Relu' id:3364 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:42.306595: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_add/add' id:3445 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_0_bn/cond/Identity, conv5_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:43.572133: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_add/add' id:3445 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_0_conv/BiasAdd:0, conv5_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:44.796112: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_1_relu/Relu' id:3517 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:46.258673: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_2_relu/Relu' id:3588 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:47.485528: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_add/add' id:3669 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_out/Relu, conv5_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:48.566980: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_1_relu/Relu' id:3741 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:50.032718: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_2_relu/Relu' id:3812 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:51.098022: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_add/add' id:3893 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_add/add}} = AddV2[T=DT_FLOAT](conv5_block2_out/Relu, conv5_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-03_09:00:51.304753/temp
2023-11-03 09:00:52,618 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-03_09:00:51.304753/temp
INFO:tensorflow:Restoring parameters from ./replace_relu6_with_relu_2023-11-03_09:00:53.247088/temp
2023-11-03 09:00:54,638 - tensorflow - INFO - Restoring parameters from ./replace_relu6_with_relu_2023-11-03_09:00:53.247088/temp
INFO:tensorflow:Restoring parameters from ./temp_cls_2023-11-03_09:01:30.907458/temp
2023-11-03 09:01:33,173 - tensorflow - INFO - Restoring parameters from ./temp_cls_2023-11-03_09:01:30.907458/temp
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[3], line 19
15 # Equalize a model with Batchnorms
16 # Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
17 # use the new session returned for further evaluations on TF graph
18 with sess.as_default():
---> 19 new_session = equalize_model(sess, input_op_name, output_op_name)
20 tf_tensorboard_writer = tf.compat.v1.summary.FileWriter('graphs_cle2', new_session.graph)
21 # op_new=new_session.graph.get_operations()
22 # print(op_new[2])
23 # [print(m.values()) for m in op_new]
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:853, in equalize_model(sess, start_op_names, output_op_names)
849 after_cls_sess, cls_set_info_list = CrossLayerScaling.scale_model(after_relu_replace_sess, start_op_names,
850 output_op_names)
852 # high-bias fold
--> 853 after_hbf_sess = HighBiasFold.bias_fold(after_cls_sess, folded_pairs, cls_set_info_list)
855 return after_hbf_sess
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:745, in HighBiasFold.bias_fold(sess, folded_pairs, cls_set_info_list)
732 """
733 Folds bias values greater than 3 * sigma to next layer's bias
734
(...)
739
740 """
742 with sess.graph.as_default():
743
744 # refresh the references saved during bn fold and cls.
--> 745 cls_set_info_list, bn_layers = HighBiasFold._refresh_layer_set_info_before_hbf(sess, folded_pairs,
746 cls_set_info_list)
748 if not bn_layers:
749 logger.error('High Bias folding is not supported for models without BatchNorm Layers')
File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:721, in HighBiasFold._refresh_layer_set_info_before_hbf(sess, folded_pairs, cls_set_info_list)
718 # update info saved during batchnorm fold
719 for conv_bn in folded_pairs:
720 # get the new op ref from it's name
--> 721 bn_dict[conv_bn[0].name] = dict_names_to_tf_ops[conv_bn[1].name]
723 # update info saved during cls
724 ClsSetInfo.map_cls_sets_to_new_session(dict_names_to_tf_ops, cls_set_info_list)
KeyError: 'conv1_bn/cond/Identity'
can you please kindly check the Key error, i tried changing in cross_layer_equalization.py
file but still the error persists. Thank you in advance.
@sandeep1404 this definitely looks like a bug. let me take a look. Thanks for reporting this.
Thank you @quic-hitameht for the response, suggest me the solution once if you fixed it. Thank you in advance.
Hi @sandeep1404, I am getting an error at Batchnorm same as yours. Have you solve this bug yet? If yes, please guide me how to solve it.
Hello everyone, I started exploring AIMET library and I installed AIMET using docker, when i started trying to implement CLE from the documentation https://quic.github.io/aimet-pages/AimetDocs/api_docs/tensorflow_cross_layer_equalization.html#tf-cle . I am using the resnet50 model for 10 class classification. I am getting errors while calling the function
cross_layer_equalization_auto()
. My code is given below:My error when i call the fucntion is shown below:
I am unable to interpret this error can anyone help me in understanding what is the problem and how to correct the issue, that would be a great help for me. Thank you in advance.