search

quic / aimet

AIMET is a library that provides advanced quantization and compression techniques for trained neural network models.
https://quic.github.io/aimet-pages/index.html
Other
2.08k stars 373 forks source link

Error while using AIMET TensorFlow Cross Layer Equalization APIs #2543

Open sandeep1404 opened 10 months ago

sandeep1404 commented 10 months ago

Hello everyone, I started exploring AIMET library and I installed AIMET using docker, when i started trying to implement CLE from the documentation https://quic.github.io/aimet-pages/AimetDocs/api_docs/tensorflow_cross_layer_equalization.html#tf-cle . I am using the resnet50 model for 10 class classification. I am getting errors while calling the function cross_layer_equalization_auto() . My code is given below:

def cross_layer_equalization_auto():
    """ perform auto cross layer equalization """

    # load a model
    tf.keras.backend.clear_session()
    model_resnet = ResNet50(weights='imagenet', input_shape=(224, 224, 3),include_top=False)
    x=model_resnet.output
    x=tf.keras.layers.GlobalAveragePooling2D()(x)
    pred=tf.keras.layers.Dense(10,activation='softmax')(x)

    model_f=tf.keras.Model(inputs=model_resnet.input, outputs=pred)
    # model_f.summary()

    sess = tf.compat.v1.Session()
    op = sess.graph.get_operations()

    # get starting op name to invoke api for cle
    input_op_name = 'input_1'
    output_op_name = 'dense/Softmax'

    # Equalize a model with Batchnorms
    # Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
    # use the new session returned for further evaluations on TF graph
    with sess.as_default():
        sess.run(tf.global_variables_initializer())
        new_session = equalize_model(sess, input_op_name, output_op_name)
    sess.close()

My error when i call the fucntion is shown below:

2023-11-02 07:09:41,995 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,996 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,996 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,997 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,997 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,998 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,998 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:41,999 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:41,999 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,000 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,000 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,001 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,001 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,001 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,002 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,002 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,003 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,003 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,004 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,004 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv2_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,005 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,006 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,006 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,006 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,007 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,007 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,007 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,008 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,008 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,009 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,009 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,009 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,010 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,010 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,010 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,011 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,011 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,012 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,012 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,012 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block4_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,013 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,013 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block4_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,013 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,014 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv3_block4_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,014 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,014 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,015 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,015 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,015 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,016 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,016 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,017 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,017 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,018 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,018 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,018 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,019 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,019 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,020 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,022 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,023 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,023 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,024 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,024 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,024 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block4_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block4_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,025 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,026 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block4_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,026 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,027 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block5_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,027 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,028 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block5_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,028 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,029 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block5_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,029 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,029 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block6_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,030 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,030 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block6_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,031 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,031 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv4_block6_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,031 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,032 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_0_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,033 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,033 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,034 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,035 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,035 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block1_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block2_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,036 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,037 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block2_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,037 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,037 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block2_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block3_1_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,038 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block3_2_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,039 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42,039 - ConnectedGraph - WARNING - Unable to identify module name, using name of first op as module name: conv5_block3_3_bn/cond/FusedBatchNormV3_1
2023-11-02 07:09:42,039 - ConnectedGraph - WARNING - Unable to identify associated op of module, setting first op in ops_list as associated op.
2023-11-02 07:09:42.732657: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.732804: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.732866: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.732964: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.733024: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-02 07:09:42.733077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 648 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-02_07:09:42.252434/temp
2023-11-02 07:09:43,125 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-02_07:09:42.252434/temp
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1378, in BaseSession._do_call(self, fn, *args)
   1377 try:
-> 1378   return fn(*args)
   1379 except errors.OpError as e:

File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1361, in BaseSession._do_run.<locals>._run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1360 self._extend_graph()
-> 1361 return self._call_tf_sessionrun(options, feed_dict, fetch_list,
   1362                                 target_list, run_metadata)

File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1454, in BaseSession._call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata)
   1452 def _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list,
   1453                         run_metadata):
-> 1454   return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
   1455                                           fetch_list, target_list,
   1456                                           run_metadata)

InvalidArgumentError: 2 root error(s) found.
  (0) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
     [[{{node Placeholder_3}}]]
     [[ReadVariableOp_3/_641]]
  (1) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
     [[{{node Placeholder_3}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
Cell In[8], line 1
----> 1 cross_layer_equalization_auto()

Cell In[7], line 31, in cross_layer_equalization_auto()
     29 with sess.as_default():
     30     sess.run(tf.global_variables_initializer())
---> 31     new_session = equalize_model(sess, input_op_name, output_op_name)
     32 sess.close()

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:842, in equalize_model(sess, start_op_names, output_op_names)
    839     start_op_names = [start_op_names]
    841 # fold batchnorm layers
--> 842 after_bn_fold_sess, folded_pairs = fold_all_batch_norms(sess, start_op_names, output_op_names)
    844 # replace any ReLU6 layers with ReLU
    845 graph_util = GraphSearchUtils(after_bn_fold_sess.graph, start_op_names, output_op_names)

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:364, in fold_all_batch_norms(sess, input_op_names, output_op_names)
    361     pairs_to_return.append((pair[0], pair[1].op))
    363 # Convert the standalone BNs which are not folded
--> 364 bn_converted = convert_standalone_batchnorms(after_fold_sess, input_op_names, output_op_names, bns_to_fold)
    365 if bn_converted:
    366     logger.info("%d BatchNorms' weights got converted", len(bn_converted))

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:394, in convert_standalone_batchnorms(sess, input_op_names, output_op_names, bns_folded)
    392 for op in list_of_ordered_ops:
    393     if op.type in ['FusedBatchNormV3', 'FusedBatchNorm', 'BatchNormalization'] and op not in bns_folded:
--> 394         convert_batchnorm_parameters(sess, op)
    395         converted_bns.append(op)
    396         logger.debug("%s weights got converted", op)

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:407, in convert_batchnorm_parameters(sess, op)
    400 def convert_batchnorm_parameters(sess, op):
    401     """
    402     Convert the weights of BN such that it works as y = weights * x + bias
    403 
    404     :param sess: TF Session in which the graph is loaded
    405     :param op: bn_op which whose weights need to be converted
    406     """
--> 407     bn_params = _get_bn_params(sess, op)
    408     weight = np.array(bn_params.gamma) / np.array(bn_params.runningVar)
    409     bias = np.array(bn_params.beta) - np.array(bn_params.runningMean) * weight

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/batch_norm_fold.py:211, in _get_bn_params(sess, bn)
    208 with sess.graph.as_default():
    209     # create BNParams type and populate
    210     bn_params = libpymo.BNParams()
--> 211     bn_params.beta = BNUtils.get_beta_as_numpy_data(sess, bn).reshape(-1)
    212     bn_params.gamma = BNUtils.get_gamma_as_numpy_data(sess, bn).reshape(-1)
    213     bn_params.runningMean = BNUtils.get_moving_mean_as_numpy_data(sess, bn).reshape(-1)

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/utils/op/fusedbatchnorm.py:267, in BNUtils.get_beta_as_numpy_data(sess, bn_op)
    264 beta_tensor = BNUtils.get_beta_read_var_op_tensor(sess.graph, bn_op)
    266 with sess.graph.as_default():
--> 267     numpy_data = sess.run(beta_tensor)
    269 return numpy_data

File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:968, in BaseSession.run(self, fetches, feed_dict, options, run_metadata)
    965 run_metadata_ptr = tf_session.TF_NewBuffer() if run_metadata else None
    967 try:
--> 968   result = self._run(None, fetches, feed_dict, options_ptr,
    969                      run_metadata_ptr)
    970   if run_metadata:
    971     proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1191, in BaseSession._run(self, handle, fetches, feed_dict, options, run_metadata)
   1188 # We only want to really perform the run if fetches or targets are provided,
   1189 # or if the call is a partial run that specifies feeds.
   1190 if final_fetches or final_targets or (handle and feed_dict_tensor):
-> 1191   results = self._do_run(handle, final_targets, final_fetches,
   1192                          feed_dict_tensor, options, run_metadata)
   1193 else:
   1194   results = []

File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1371, in BaseSession._do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1368   return self._call_tf_sessionprun(handle, feed_dict, fetch_list)
   1370 if handle is None:
-> 1371   return self._do_call(_run_fn, feeds, fetches, targets, options,
   1372                        run_metadata)
   1373 else:
   1374   return self._do_call(_prun_fn, handle, feeds, fetches)

File /usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py:1397, in BaseSession._do_call(self, fn, *args)
   1392 if 'only supports NHWC tensor format' in message:
   1393   message += ('\nA possible workaround: Try disabling Grappler optimizer'
   1394               '\nby modifying the config for creating the session eg.'
   1395               '\nsession_config.graph_options.rewrite_options.'
   1396               'disable_meta_optimizer = True')
-> 1397 raise type(e)(node_def, op, message)

InvalidArgumentError: Graph execution error:

Detected at node 'Placeholder_3' defined at (most recent call last):
Node: 'Placeholder_3'
Detected at node 'Placeholder_3' defined at (most recent call last):
Node: 'Placeholder_3'
2 root error(s) found.
  (0) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
     [[{{node Placeholder_3}}]]
     [[ReadVariableOp_3/_641]]
  (1) INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder_3' with dtype float and shape [?]
     [[{{node Placeholder_3}}]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'Placeholder_3':

I am unable to interpret this error can anyone help me in understanding what is the problem and how to correct the issue, that would be a great help for me. Thank you in advance.

quic-hitameht commented 10 months ago

Hi @sandeep1404 In TF 2.x, eager execution is enabled by default and you are using AIMET TensorFlow CLE API designed for Graph mode (uses tf.Session/tf.Graph). You will need to disable eager execution in order to use it. To disable eager execution, please put tf.compat.v1.disable_eager_excution() API before instantiating the model.

import tensorflow as tf
tf.compat.v1.disable_eager_execution()

Also, if you are interested in using TF eager execution mode, you can also take a look at AIMET Keras CLE API as well which is designed for eager execution.
https://quic.github.io/aimet-pages/releases/latest/api_docs/keras_cross_layer_equalization.html#api-keras-cle

sandeep1404 commented 10 months ago

@quic-hitameht , Thanks for the reply but I already disabled eager execution and run the code, after disabling eager execution only i am getting the above error. Can you please check that once. Thank you in advance.

quic-hitameht commented 10 months ago

@sandeep1404 Could you please share your complete script to reproduce the issue? Thank you.

sandeep1404 commented 10 months ago

Hi @quic-hitameht , Thanks for the response, PFB my code the code is running fine but i cannot able to understand whats going inside the CLE, I tried to interpret the logs but i couldnt get information from it, I tried visualizing the new session graph in tensorboard, but the graph_cle folder is being created for the new session and when i open tensorboard with the logdir as graph_cle using the command tensorboard --logdir=graphs_cle it gives the following logs and when i open the localhost link http://localhost:6006/ it says This site can’t be reached

2023-11-03 05:47:16.784452: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-03 05:47:16.851571: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-11-03 05:47:16.868884: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2023-11-03 05:47:17.148442: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-11-03 05:47:17.148500: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-11-03 05:47:17.148503: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-11-03 05:47:17.526458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 05:47:17.531403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 05:47:17.531544: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
W1103 05:47:17.557937 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1103 05:47:17.559506 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.565727 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1103 05:47:17.568787 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.570783 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1103 05:47:17.572910 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.574856 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1103 05:47:17.575885 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.577856 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1103 05:47:17.578965 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
W1103 05:47:17.580727 139798600681216 plugin_event_accumulator.py:323] Found more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events.  Overwriting the graph with the newest event.
W1103 05:47:17.582290 139798600681216 plugin_event_accumulator.py:335] Found more than one metagraph event per run. Overwriting the metagraph with the newest event.
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.10.1 at http://localhost:6006/ (Press CTRL+C to quit)

can you please kindly check the code and report whats the issue and how to understand whats going inside. Thankyou in advance.

Here is my code 

# tensorflow
import tensorflow as tf
tf.compat.v1.disable_eager_execution()

from tensorflow.keras.applications.resnet50 import ResNet50

# Cross layer Equalization related imports
from aimet_tensorflow.cross_layer_equalization import equalize_model

# def cross_layer_equalization_auto():
""" perform auto cross layer equalization """

# load a model
tf.compat.v1.keras.backend.clear_session()
model_resnet = ResNet50(weights='imagenet', input_shape=(224, 224, 3))
sess = tf.compat.v1.keras.backend.get_session()
# op=sess.graph.get_operations()
#  print(op[-1])

# get starting op name to invoke api for cle
input_op_name = 'input_1'
output_op_name = 'init'

# Equalize a model with Batchnorms
# Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
# use the new session returned for further evaluations on TF graph
with sess.as_default():
    new_session = equalize_model(sess, input_op_name, output_op_name)
    tf_tensorboard_writer = tf.compat.v1.summary.FileWriter('graphs_cle', new_session.graph)
    # op_new=new_session.graph.get_operations()
    # print(op_new[2])
    # [print(m.values()) for m in op_new]

tf_tensorboard_writer.close()
sess.close()

Here are the logs of the outputs:

2023-11-03 04:56:41.819730: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-03 04:56:41.821519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:41.821685: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:41.821770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.178780: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179263: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179428: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:42.179563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
2023-11-03 04:56:42.268941: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:354] MLIR V1 optimization pass is not enabled
WARNING:tensorflow:From /tmp/ipykernel_3416/1158554977.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

2023-11-03 04:56:43,737 - tensorflow - WARNING - From /tmp/ipykernel_3416/1158554977.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

2023-11-03 04:56:44.702115: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702278: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702340: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702475: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:44.702617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-03_04:56:43.974220/temp
2023-11-03 04:56:45,042 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-03_04:56:43.974220/temp
2023-11-03 04:56:46.537949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538108: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538171: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538292: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538357: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:46.538413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./replace_relu6_with_relu_2023-11-03_04:56:45.693136/temp
2023-11-03 04:56:46,867 - tensorflow - INFO - Restoring parameters from ./replace_relu6_with_relu_2023-11-03_04:56:45.693136/temp
2023-11-03 04:56:48.083233: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083843: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083919: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:980] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-11-03 04:56:48.083991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1973 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3050 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
INFO:tensorflow:Restoring parameters from ./temp_cls_2023-11-03_04:56:47.416744/temp
2023-11-03 04:56:48,633 - tensorflow - INFO - Restoring parameters from ./temp_cls_2023-11-03_04:56:47.416744/temp
2023-11-03 04:56:48,984 - CrossLayerEqualization - ERROR - High Bias folding is not supported for models without BatchNorm Layers
quic-hitameht commented 10 months ago

@sandeep1404 Thanks for sharing your script. equalize_model API expects list of input and output names of given model. In your script, you have provided incorrect output_op_name. For tf.keras.applications.ResNet50(), it should be predictions/Softmax.

Regarding your question to understand CLE, the feature attempts to equalize weight ranges across different channels. It is beneficial for models where weight ranges for different channels in a layer has wide variance. You can use the returned session for further evaluations on TF graph.

sandeep1404 commented 10 months ago

Hi @quic-hitameht , I changed the output_op_name to predictions/Softmax but when i run the code it has some key error, here are the logs when i change the output name:

WARNING:tensorflow:From /tmp/ipykernel_8450/2048079406.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

2023-11-03 09:00:06,897 - tensorflow - WARNING - From /tmp/ipykernel_8450/2048079406.py:7: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

2023-11-03 09:00:07,116 - Utils - INFO - ...... subset to store [conv1_conv/Conv2D, conv1_bn]
2023-11-03 09:00:07,116 - Utils - INFO - ...... subset to store [conv1_conv/Conv2D, conv1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_1_conv/Conv2D, conv2_block1_1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_1_conv/Conv2D, conv2_block1_1_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_2_conv/Conv2D, conv2_block1_2_bn]
2023-11-03 09:00:07,117 - Utils - INFO - ...... subset to store [conv2_block1_2_conv/Conv2D, conv2_block1_2_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv2_block1_3_conv/Conv2D, conv2_block1_3_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv2_block1_3_conv/Conv2D, conv2_block1_3_bn]
2023-11-03 09:00:07,118 - Utils - INFO - ...... subset to store [conv3_block1_1_conv/Conv2D, conv3_block1_1_bn]
2023-11-03 09:00:07,119 - Utils - INFO - ...... subset to store [conv3_block1_1_conv/Conv2D, conv3_block1_1_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_2_conv/Conv2D, conv3_block1_2_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_2_conv/Conv2D, conv3_block1_2_bn]
2023-11-03 09:00:07,120 - Utils - INFO - ...... subset to store [conv3_block1_3_conv/Conv2D, conv3_block1_3_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv3_block1_3_conv/Conv2D, conv3_block1_3_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_1_conv/Conv2D, conv4_block1_1_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_1_conv/Conv2D, conv4_block1_1_bn]
2023-11-03 09:00:07,121 - Utils - INFO - ...... subset to store [conv4_block1_2_conv/Conv2D, conv4_block1_2_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_2_conv/Conv2D, conv4_block1_2_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_3_conv/Conv2D, conv4_block1_3_bn]
2023-11-03 09:00:07,122 - Utils - INFO - ...... subset to store [conv4_block1_3_conv/Conv2D, conv4_block1_3_bn]
2023-11-03 09:00:07,123 - Utils - INFO - ...... subset to store [conv5_block1_1_conv/Conv2D, conv5_block1_1_bn]
2023-11-03 09:00:07,123 - Utils - INFO - ...... subset to store [conv5_block1_1_conv/Conv2D, conv5_block1_1_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_2_conv/Conv2D, conv5_block1_2_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_2_conv/Conv2D, conv5_block1_2_bn]
2023-11-03 09:00:07,124 - Utils - INFO - ...... subset to store [conv5_block1_3_conv/Conv2D, conv5_block1_3_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block1_3_conv/Conv2D, conv5_block1_3_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block3_1_conv/Conv2D, conv5_block3_1_bn]
2023-11-03 09:00:07,125 - Utils - INFO - ...... subset to store [conv5_block3_1_conv/Conv2D, conv5_block3_1_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_2_conv/Conv2D, conv5_block3_2_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_2_conv/Conv2D, conv5_block3_2_bn]
2023-11-03 09:00:07,126 - Utils - INFO - ...... subset to store [conv5_block3_3_conv/Conv2D, conv5_block3_3_bn]
2023-11-03 09:00:07,127 - Utils - INFO - ...... subset to store [conv5_block2_1_conv/Conv2D, conv5_block2_1_bn]
2023-11-03 09:00:07,127 - Utils - INFO - ...... subset to store [conv5_block2_1_conv/Conv2D, conv5_block2_1_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_2_conv/Conv2D, conv5_block2_2_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_2_conv/Conv2D, conv5_block2_2_bn]
2023-11-03 09:00:07,128 - Utils - INFO - ...... subset to store [conv5_block2_3_conv/Conv2D, conv5_block2_3_bn]
2023-11-03 09:00:07,129 - Utils - INFO - ...... subset to store [conv5_block1_0_conv/Conv2D, conv5_block1_0_bn]
2023-11-03 09:00:07,129 - Utils - INFO - ...... subset to store [conv4_block6_1_conv/Conv2D, conv4_block6_1_bn]
2023-11-03 09:00:07,130 - Utils - INFO - ...... subset to store [conv4_block6_1_conv/Conv2D, conv4_block6_1_bn]
2023-11-03 09:00:07,133 - Utils - INFO - ...... subset to store [conv4_block6_2_conv/Conv2D, conv4_block6_2_bn]
2023-11-03 09:00:07,133 - Utils - INFO - ...... subset to store [conv4_block6_2_conv/Conv2D, conv4_block6_2_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block6_3_conv/Conv2D, conv4_block6_3_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block5_1_conv/Conv2D, conv4_block5_1_bn]
2023-11-03 09:00:07,134 - Utils - INFO - ...... subset to store [conv4_block5_1_conv/Conv2D, conv4_block5_1_bn]
2023-11-03 09:00:07,135 - Utils - INFO - ...... subset to store [conv4_block5_2_conv/Conv2D, conv4_block5_2_bn]
2023-11-03 09:00:07,135 - Utils - INFO - ...... subset to store [conv4_block5_2_conv/Conv2D, conv4_block5_2_bn]
2023-11-03 09:00:07,136 - Utils - INFO - ...... subset to store [conv4_block5_3_conv/Conv2D, conv4_block5_3_bn]
2023-11-03 09:00:07,136 - Utils - INFO - ...... subset to store [conv4_block4_1_conv/Conv2D, conv4_block4_1_bn]
2023-11-03 09:00:07,137 - Utils - INFO - ...... subset to store [conv4_block4_1_conv/Conv2D, conv4_block4_1_bn]
2023-11-03 09:00:07,137 - Utils - INFO - ...... subset to store [conv4_block4_2_conv/Conv2D, conv4_block4_2_bn]
2023-11-03 09:00:07,138 - Utils - INFO - ...... subset to store [conv4_block4_2_conv/Conv2D, conv4_block4_2_bn]
2023-11-03 09:00:07,138 - Utils - INFO - ...... subset to store [conv4_block4_3_conv/Conv2D, conv4_block4_3_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_1_conv/Conv2D, conv4_block3_1_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_1_conv/Conv2D, conv4_block3_1_bn]
2023-11-03 09:00:07,139 - Utils - INFO - ...... subset to store [conv4_block3_2_conv/Conv2D, conv4_block3_2_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block3_2_conv/Conv2D, conv4_block3_2_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block3_3_conv/Conv2D, conv4_block3_3_bn]
2023-11-03 09:00:07,140 - Utils - INFO - ...... subset to store [conv4_block2_1_conv/Conv2D, conv4_block2_1_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_1_conv/Conv2D, conv4_block2_1_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_2_conv/Conv2D, conv4_block2_2_bn]
2023-11-03 09:00:07,141 - Utils - INFO - ...... subset to store [conv4_block2_2_conv/Conv2D, conv4_block2_2_bn]
2023-11-03 09:00:07,142 - Utils - INFO - ...... subset to store [conv4_block2_3_conv/Conv2D, conv4_block2_3_bn]
2023-11-03 09:00:07,143 - Utils - INFO - ...... subset to store [conv4_block1_0_conv/Conv2D, conv4_block1_0_bn]
2023-11-03 09:00:07,143 - Utils - INFO - ...... subset to store [conv3_block4_1_conv/Conv2D, conv3_block4_1_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_1_conv/Conv2D, conv3_block4_1_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_2_conv/Conv2D, conv3_block4_2_bn]
2023-11-03 09:00:07,144 - Utils - INFO - ...... subset to store [conv3_block4_2_conv/Conv2D, conv3_block4_2_bn]
2023-11-03 09:00:07,145 - Utils - INFO - ...... subset to store [conv3_block4_3_conv/Conv2D, conv3_block4_3_bn]
2023-11-03 09:00:07,146 - Utils - INFO - ...... subset to store [conv3_block3_1_conv/Conv2D, conv3_block3_1_bn]
2023-11-03 09:00:07,146 - Utils - INFO - ...... subset to store [conv3_block3_1_conv/Conv2D, conv3_block3_1_bn]
2023-11-03 09:00:07,147 - Utils - INFO - ...... subset to store [conv3_block3_2_conv/Conv2D, conv3_block3_2_bn]
2023-11-03 09:00:07,147 - Utils - INFO - ...... subset to store [conv3_block3_2_conv/Conv2D, conv3_block3_2_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block3_3_conv/Conv2D, conv3_block3_3_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block2_1_conv/Conv2D, conv3_block2_1_bn]
2023-11-03 09:00:07,148 - Utils - INFO - ...... subset to store [conv3_block2_1_conv/Conv2D, conv3_block2_1_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_2_conv/Conv2D, conv3_block2_2_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_2_conv/Conv2D, conv3_block2_2_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block2_3_conv/Conv2D, conv3_block2_3_bn]
2023-11-03 09:00:07,149 - Utils - INFO - ...... subset to store [conv3_block1_0_conv/Conv2D, conv3_block1_0_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_1_conv/Conv2D, conv2_block3_1_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_1_conv/Conv2D, conv2_block3_1_bn]
2023-11-03 09:00:07,150 - Utils - INFO - ...... subset to store [conv2_block3_2_conv/Conv2D, conv2_block3_2_bn]
2023-11-03 09:00:07,151 - Utils - INFO - ...... subset to store [conv2_block3_2_conv/Conv2D, conv2_block3_2_bn]
2023-11-03 09:00:07,151 - Utils - INFO - ...... subset to store [conv2_block3_3_conv/Conv2D, conv2_block3_3_bn]
2023-11-03 09:00:07,152 - Utils - INFO - ...... subset to store [conv2_block2_1_conv/Conv2D, conv2_block2_1_bn]
2023-11-03 09:00:07,152 - Utils - INFO - ...... subset to store [conv2_block2_1_conv/Conv2D, conv2_block2_1_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_2_conv/Conv2D, conv2_block2_2_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_2_conv/Conv2D, conv2_block2_2_bn]
2023-11-03 09:00:07,153 - Utils - INFO - ...... subset to store [conv2_block2_3_conv/Conv2D, conv2_block2_3_bn]
2023-11-03 09:00:07,154 - Utils - INFO - ...... subset to store [conv2_block1_0_conv/Conv2D, conv2_block1_0_bn]
2023-11-03 09:00:07.588003: W tensorflow/c/c_api.cc:349] Operation '{name:'conv1_relu/Relu' id:77 op device:{requested: '', assigned: ''} def:{{{node conv1_relu/Relu}} = Relu[T=DT_FLOAT](conv1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:08.147625: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_1_relu/Relu' id:221 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:08.936135: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_2_relu/Relu' id:292 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:09.711713: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_add/add' id:363 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_0_bn/cond/Identity, conv2_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:10.443266: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block1_add/add' id:363 op device:{requested: '', assigned: ''} def:{{{node conv2_block1_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_0_conv/BiasAdd:0, conv2_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:11.163123: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_1_relu/Relu' id:435 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:11.954976: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_2_relu/Relu' id:506 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:12.688118: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block2_add/add' id:577 op device:{requested: '', assigned: ''} def:{{{node conv2_block2_add/add}} = AddV2[T=DT_FLOAT](conv2_block1_out/Relu, conv2_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:13.299646: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_1_relu/Relu' id:649 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:13.896197: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_2_relu/Relu' id:720 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv2_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:14.492094: W tensorflow/c/c_api.cc:349] Operation '{name:'conv2_block3_add/add' id:791 op device:{requested: '', assigned: ''} def:{{{node conv2_block3_add/add}} = AddV2[T=DT_FLOAT](conv2_block2_out/Relu, conv2_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:15.089717: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_1_relu/Relu' id:933 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:15.713069: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_2_relu/Relu' id:1004 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:16.363501: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_add/add' id:1075 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_0_bn/cond/Identity, conv3_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:17.035267: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block1_add/add' id:1075 op device:{requested: '', assigned: ''} def:{{{node conv3_block1_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_0_conv/BiasAdd:0, conv3_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:17.673926: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_1_relu/Relu' id:1147 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:18.419134: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_2_relu/Relu' id:1218 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:19.247085: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block2_add/add' id:1289 op device:{requested: '', assigned: ''} def:{{{node conv3_block2_add/add}} = AddV2[T=DT_FLOAT](conv3_block1_out/Relu, conv3_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:20.006269: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_1_relu/Relu' id:1361 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:20.701548: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_2_relu/Relu' id:1432 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:21.338198: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block3_add/add' id:1503 op device:{requested: '', assigned: ''} def:{{{node conv3_block3_add/add}} = AddV2[T=DT_FLOAT](conv3_block2_out/Relu, conv3_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:22.048276: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_1_relu/Relu' id:1575 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_1_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block4_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:22.755388: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_2_relu/Relu' id:1646 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_2_relu/Relu}} = Relu[T=DT_FLOAT](conv3_block4_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:23.630489: W tensorflow/c/c_api.cc:349] Operation '{name:'conv3_block4_add/add' id:1717 op device:{requested: '', assigned: ''} def:{{{node conv3_block4_add/add}} = AddV2[T=DT_FLOAT](conv3_block3_out/Relu, conv3_block4_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:24.498999: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_1_relu/Relu' id:1869 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:25.486446: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_2_relu/Relu' id:1940 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:26.267102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_add/add' id:2021 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_0_bn/cond/Identity, conv4_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:27.074924: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block1_add/add' id:2021 op device:{requested: '', assigned: ''} def:{{{node conv4_block1_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_0_conv/BiasAdd:0, conv4_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:27.814852: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_1_relu/Relu' id:2093 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:28.713874: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_2_relu/Relu' id:2164 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:29.490330: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block2_add/add' id:2245 op device:{requested: '', assigned: ''} def:{{{node conv4_block2_add/add}} = AddV2[T=DT_FLOAT](conv4_block1_out/Relu, conv4_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:30.251893: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_1_relu/Relu' id:2317 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:31.101687: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_2_relu/Relu' id:2388 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:31.850806: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block3_add/add' id:2469 op device:{requested: '', assigned: ''} def:{{{node conv4_block3_add/add}} = AddV2[T=DT_FLOAT](conv4_block2_out/Relu, conv4_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:32.613102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_1_relu/Relu' id:2541 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block4_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:33.422959: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_2_relu/Relu' id:2612 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block4_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:34.179514: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block4_add/add' id:2693 op device:{requested: '', assigned: ''} def:{{{node conv4_block4_add/add}} = AddV2[T=DT_FLOAT](conv4_block3_out/Relu, conv4_block4_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:34.946318: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_1_relu/Relu' id:2765 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block5_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:35.775102: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_2_relu/Relu' id:2836 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block5_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:36.532354: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block5_add/add' id:2917 op device:{requested: '', assigned: ''} def:{{{node conv4_block5_add/add}} = AddV2[T=DT_FLOAT](conv4_block4_out/Relu, conv4_block5_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:37.313127: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_1_relu/Relu' id:2989 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_1_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block6_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:38.166129: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_2_relu/Relu' id:3060 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_2_relu/Relu}} = Relu[T=DT_FLOAT](conv4_block6_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:38.963474: W tensorflow/c/c_api.cc:349] Operation '{name:'conv4_block6_add/add' id:3141 op device:{requested: '', assigned: ''} def:{{{node conv4_block6_add/add}} = AddV2[T=DT_FLOAT](conv4_block5_out/Relu, conv4_block6_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:39.889524: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_1_relu/Relu' id:3293 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block1_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:41.284167: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_2_relu/Relu' id:3364 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block1_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:42.306595: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_add/add' id:3445 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_0_bn/cond/Identity, conv5_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:43.572133: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block1_add/add' id:3445 op device:{requested: '', assigned: ''} def:{{{node conv5_block1_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_0_conv/BiasAdd:0, conv5_block1_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:44.796112: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_1_relu/Relu' id:3517 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block2_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:46.258673: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_2_relu/Relu' id:3588 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block2_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:47.485528: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block2_add/add' id:3669 op device:{requested: '', assigned: ''} def:{{{node conv5_block2_add/add}} = AddV2[T=DT_FLOAT](conv5_block1_out/Relu, conv5_block2_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:48.566980: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_1_relu/Relu' id:3741 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_1_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block3_1_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:50.032718: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_2_relu/Relu' id:3812 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_2_relu/Relu}} = Relu[T=DT_FLOAT](conv5_block3_2_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
2023-11-03 09:00:51.098022: W tensorflow/c/c_api.cc:349] Operation '{name:'conv5_block3_add/add' id:3893 op device:{requested: '', assigned: ''} def:{{{node conv5_block3_add/add}} = AddV2[T=DT_FLOAT](conv5_block2_out/Relu, conv5_block3_3_conv/BiasAdd:0)}}' was changed by updating input tensor after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
INFO:tensorflow:Restoring parameters from ./temp_bn_fold_2023-11-03_09:00:51.304753/temp
2023-11-03 09:00:52,618 - tensorflow - INFO - Restoring parameters from ./temp_bn_fold_2023-11-03_09:00:51.304753/temp
INFO:tensorflow:Restoring parameters from ./replace_relu6_with_relu_2023-11-03_09:00:53.247088/temp
2023-11-03 09:00:54,638 - tensorflow - INFO - Restoring parameters from ./replace_relu6_with_relu_2023-11-03_09:00:53.247088/temp
INFO:tensorflow:Restoring parameters from ./temp_cls_2023-11-03_09:01:30.907458/temp
2023-11-03 09:01:33,173 - tensorflow - INFO - Restoring parameters from ./temp_cls_2023-11-03_09:01:30.907458/temp
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[3], line 19
     15 # Equalize a model with Batchnorms
     16 # Performs BatchNorm fold, replacing Relu6 with Relu, Cross layer scaling and High bias fold
     17 # use the new session returned for further evaluations on TF graph
     18 with sess.as_default():
---> 19     new_session = equalize_model(sess, input_op_name, output_op_name)
     20     tf_tensorboard_writer = tf.compat.v1.summary.FileWriter('graphs_cle2', new_session.graph)
     21     # op_new=new_session.graph.get_operations()
     22     # print(op_new[2])
     23     # [print(m.values()) for m in op_new]

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:853, in equalize_model(sess, start_op_names, output_op_names)
    849 after_cls_sess, cls_set_info_list = CrossLayerScaling.scale_model(after_relu_replace_sess, start_op_names,
    850                                                                   output_op_names)
    852 # high-bias fold
--> 853 after_hbf_sess = HighBiasFold.bias_fold(after_cls_sess, folded_pairs, cls_set_info_list)
    855 return after_hbf_sess

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:745, in HighBiasFold.bias_fold(sess, folded_pairs, cls_set_info_list)
    732 """
    733 Folds bias values greater than 3 * sigma to next layer's bias
    734 
   (...)
    739 
    740 """
    742 with sess.graph.as_default():
    743 
    744     # refresh the references saved during bn fold and cls.
--> 745     cls_set_info_list, bn_layers = HighBiasFold._refresh_layer_set_info_before_hbf(sess, folded_pairs,
    746                                                                                    cls_set_info_list)
    748     if not bn_layers:
    749         logger.error('High Bias folding is not supported for models without BatchNorm Layers')

File /usr/local/lib/python3.8/dist-packages/aimet_tensorflow/cross_layer_equalization.py:721, in HighBiasFold._refresh_layer_set_info_before_hbf(sess, folded_pairs, cls_set_info_list)
    718 # update info saved during batchnorm fold
    719 for conv_bn in folded_pairs:
    720     # get the new op ref from it's name
--> 721     bn_dict[conv_bn[0].name] = dict_names_to_tf_ops[conv_bn[1].name]
    723 # update info saved during cls
    724 ClsSetInfo.map_cls_sets_to_new_session(dict_names_to_tf_ops, cls_set_info_list)

KeyError: 'conv1_bn/cond/Identity'

can you please kindly check the Key error, i tried changing in cross_layer_equalization.py file but still the error persists. Thank you in advance.

quic-hitameht commented 10 months ago

@sandeep1404 this definitely looks like a bug. let me take a look. Thanks for reporting this.

sandeep1404 commented 10 months ago

Thank you @quic-hitameht for the response, suggest me the solution once if you fixed it. Thank you in advance.

hoangtv2000 commented 4 months ago

Hi @sandeep1404, I am getting an error at Batchnorm same as yours. Have you solve this bug yet? If yes, please guide me how to solve it.