Missing model annotations in TF2 quantizer

[jeinstei]

Platform: docker.io/xilinx/vitis-ai created on 2021-10-08 Environment: vitis-ai-tensorflow2

I believe this is two separate bugs, but they're both included here as I can split them out if the mods prefer.

I'm using the models below, basically:

dense (None, 37)
dropout (None, 20)
dense_1 (None, 20)
dropout_1 (None, 20)
dense_2 (None, 20)
dropout_2 (None, 10)
dense_3 (None, 10)

Removing Dropout layers to avoid bug with remove_dropout config transformation
dense (None, 37)
dense_1 (None, 20)
dense_2 (None, 20)
dense_3 (None, 10)

[VAI INFO] Start CrossLayerEqualization...
10/10 [==============================] - 0s 21ms/step
[VAI INFO] CrossLayerEqualization Done.

With the commands here:

from tensorflow_model_optimization.quantization.keras import vitis_quantize
quantizer = vitis_quantize.VitisQuantizer(model)
quantized_model = quantizer.quantize_model(calib_dataset=ds)

The quantizer process involves taking a float model, optimizing the float model through a series of transformations, and then quantizing the model. In general, this process involves applying a series of transforms to the model in preparation for quantization using one of the provided pipelines. In our case, this is the 8bit pipeline as in this file: pipeline link

  def optimize_model(self, configs={}, **kwargs):
    """Get optimized model.

    Available configs:
       * remove_dropout=True
       * fold_conv_bn=True
       * fold_bn=True
       * replace_relu6=False
       * include_cle=True
       * cle_steps=5
    """
})

If our model includes any Dropout layers, this process raises a size mismatch error, likely due to a bug in the transformation for remove_dropout. If this config is removed, then a model with Dropout layers will be "optimized", or we can rebuild our model with no Dropout layers. This allows us to complete the optimize_model step of the quantize_model method.

At this point, the quantizer should have several internal members populated-- self._optimized_model, self._layer_metadata, and self._candidate_layers. Unfortunately, self._layer_metadata and self._candidate_layers never get populated. That's only the first problem, as the quantize_model method then calls the quantize_apply method of vitis_quantize.py

Note that in the docstring of this method, it clearly states we need a model passed in that has its layers wrapped by annotations provided by quantize_annotate.

def quantize_apply(model, candidate_layers, layer_metadata, quantize_strategy,
                   mode):
  """Quantize a `tf.keras` model that has been annotated for quantization.
...
  Args:
    model: A `tf.keras` Sequential or Functional model which has been annotated
      with `quantize_annotate`. It can have pre-trained weights.

As far as I can tell, this procedure is never called for the pipeline used by quantizer, leaving us with an object with no annotated layers (example of proper annotation). I believe it should be possible to run optimize_model and then annotate the layers correctly, as well as getting the proper metadata variables filled, but I am not exactly sure yet at what step this is failing at. it was either completely forgotten, or somewhere deep down in the bowels of the ModelTransformer and some transformations.

This leads me to the exception below as a keras.layer doesn't have a name attribute, while a "properly" annotated layer wrapped in a QuantizeAnnotate wrapper does.

/workspace/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_quantize_transforms.py in replacement(self, match_layer)
    519   def replacement(self, match_layer):
    520     act_layer_node = match_layer
--> 521     act_layer_name = act_layer_node.layer['name']
    522 
    523     vitis_sigmoid_layer = vitis_activation.VitisSigmoid(name=act_layer_name +

KeyError: 'name'

This has led to a Xilinx support forum post that I'm keeping up to date as well.

[jeinstei]

After working with our FAE, a workaround is available by running a sed command:

sed -i "s/act_layer_node.layer\['name'\]/act_layer_node.layer\['config'\]\['name'\]/g" /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_quantize_transforms.py

This led to finding an additional bug with ReLU activation functions where quantize_model had to be changed due to an issue with an internal transformer:

FROM: quantized_model = quantizer.quantize_model(calib_dataset=ds)
  TO: quantized_model = quantizer.quantize_model(calib_dataset=ds, separate_conv_act=False)

[jeinstei]

Both the Dropout removal and the other issues look to be resolved in the 2.0 release. Closing.

[AfifaIshtiaq]

Both the Dropout removal and the other issues look to be resolved in the 2.0 release. Closing.

Can you please also help me figure out the following issue: It gives me the following issue. I have attached some of the files from my dataset in a .zip Dataset.zip

file and a preprocessing script preprocess.txt . Can you tell me where is the issue? My model accepts input like this in which In1,In2 In3,In4 In5,In6 are concatenated image

Issue: [VAI INFO] Update custom_layer_type: [] Traceback (most recent call last): File "Deep_RX_ptq_mod.py", line 82, in quantized_model = quantizer.quantize_model(calib_dataset=inputs) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 566, in quantize_model self.optimize_model() File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 454, in optimize_model self._create_optimized_model() File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 228, in _create_optimized_model quantize_strategy=self._quantize_strategy) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 700, in create_optimize_model model, candidate_layers, layer_metadata) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_transforms_pipeline.py", line 74, in apply model, configs, available_transforms, candidate_layers, layer_metadata) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_transforms_pipeline.py", line 50, in _apply_availables layer_metadata).recursive_transform() File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 738, in recursive_transform self.layer_metadata).transform() File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 704, in transform self._set_layer_weights(layer, weights_map) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 593, in _set_layer_weights K.batch_set_value(weight_value_tuples) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/keras/backend.py", line 4019, in batch_set_value x.assign(np.asarray(value, dtype=dtype_numpy(x))) ValueError: Cannot assign value to variable ' BN-2-0/gamma:0': Shape mismatch.The variable shape (64,), and the assigned value shape (128,) are incompatible.

[jeinstei]

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

[AfifaIshtiaq]

Tensorflow addon LAMB is used I have attached my preprocess script and dataset

nputs = [dataset_rx,dataset_tx,dataset_ch]

import tensorflow as tf # pylint: disable=g-bad-import-orde from tensorflow import keras from deep_rx_model import BitErrorRate from deep_rx_model import binary_sigmoid_cross_entropy import tensorflow_addons as tfa

load model graph without compiling it

model = tf.keras.models.load_model('deep_rx.h5', compile=False)

# recompile model with customization that were used during training

float_model=model.compile(optimizer=tfa.optimizers.LAMB(), loss=binary_sigmoid_cross_entropy,metrics=[BitErrorRate()]) model.predict(inputs) from tensorflow_model_optimization.quantization.keras import vitis_quantize quantizer = vitis_quantize.VitisQuantizer(model) quantized_model = quantizer.quantize_model(calib_dataset=inputs)

[AfifaIshtiaq]

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model? But it gives me the issue on self._create_optimized_model()

File "test_model.py", line 50, in quantized_model = quantizer.quantize_model(calib_dataset=test_generator) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 566, in quantize_model self.optimize_model() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 454, in optimize_model self._create_optimized_model() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 228, in _create_optimized_model quantize_strategy=self._quantize_strategy) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 700, in create_optimize_model model, candidate_layers, layer_metadata) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_transforms_pipeline.py", line 92, in apply layer_metadata) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_transforms_pipeline.py", line 50, in _apply_availables layer_metadata).recursive_transform() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 738, in recursive_transform self.layer_metadata).transform() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 704, in transform self._set_layer_weights(layer, weights_map) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 593, in _set_layer_weights K.batch_set_value(weight_value_tuples) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper return target(*args, **kwargs) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/backend.py", line 3775, in batch_set_value x.assign(np.asarray(value, dtype=dtype_numpy(x))) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 902, in assign (tensor_name, self._shape, value_tensor.shape)) ValueError: Cannot assign to variable BN-0-0/depthwise_kernel:0 due to variable shape (1, 1, 64, 1) and value shape (3, 3, 64, 1) are incompatible

[AfifaIshtiaq]

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model)

/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

[jeinstei]

Again -- this should be another issue. Also your dataset is (3) npz files, but I don't see a model anywhere, meaning that one won't be able to diagnose it as we don't know the model layers involved.

What version of Vitis AI?

[AfifaIshtiaq]

This is my model. I shared only three files just for reference but I hav more than 1000

Again -- this should be another issue. Also your dataset is (3) npz files, but I don't see a model anywhere, meaning that one won't be able to diagnose it as we don't know the model layers involved.

What version of Vitis AI?

[foolmarks]

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model)

/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model)

/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

That is just a warning that TF always generates - it is nothing to do with any issues that you might have.

[AfifaIshtiaq]

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

That is just a warning that TF always generates - it is nothing to do with any issues that you might have.

But I'm unable to pinpoint the issue causing this Cannot assign to variable BN-0-0/depthwise_kernel:0 due to variable shape (1, 1, 64, 1) and value shape (3, 3, 64, 1) are incompatible

[foolmarks]

Are you able to share the trained floating-point checkpoint?If yes, email me at @. Sent from my Galaxy -------- Original message --------From: AfifaIshtiaq @.> Date: 12/05/2022 14:34 (GMT+01:00) To: Xilinx/Vitis-AI @.> Cc: Mark Harvey @.>, Comment @.***> Subject: Re: [Xilinx/Vitis-AI] Missing model annotations in TF2 quantizer (Issue #628)

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model?

It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

That is just a warning that TF always generates - it is nothing to do with any issues that you might have.

But I'm unable to pinpoint the issue causing this Cannot assign to variable BN-0-0/depthwise_kernel:0 due to variable shape (1, 1, 64, 1) and value shape (3, 3, 64, 1) are incompatible

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[AfifaIshtiaq]

Are you able to share the trained floating-point checkpoint?If yes, email me at @. Sent from my Galaxy -------- Original message --------From: AfifaIshtiaq @.> Date: 12/05/2022 14:34 (GMT+01:00) To: Xilinx/Vitis-AI @.> Cc: Mark Harvey @.>, Comment @.> Subject: Re: [Xilinx/Vitis-AI] Missing model annotations in TF2 quantizer (Issue #628) This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model? It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model? It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] That is just a warning that TF always generates - it is nothing to do with any issues that you might have. But I'm unable to pinpoint the issue causing this Cannot assign to variable BN-0-0/depthwise_kernel:0 due to variable shape (1, 1, 64, 1) and value shape (3, 3, 64, 1) are incompatible —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.>

I cannot see the emaild Id

[foolmarks]

Looks like my email address was automatically removed.

[AfifaIshtiaq]

May be try uploading in a text file later On Thu 12. May 2022 at 16:53, Mark Harvey @.***> wrote:

Looks like my email address was automatically removed.

— Reply to this email directly, view it on GitHub https://github.com/Xilinx/Vitis-AI/issues/628#issuecomment-1125095275, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQIQY4WXDLZOLZXLUGG7FG3VJULQBANCNFSM5LXTCWGA . You are receiving this because you commented.Message ID: @.***>

[AfifaIshtiaq]

Are you able to share the trained floating-point checkpoint?If yes, email me at @. Sent from my Galaxy -------- Original message --------From: AfifaIshtiaq @.> Date: 12/05/2022 14:34 (GMT+01:00) To: Xilinx/Vitis-AI @.> Cc: Mark Harvey @.>, Comment @.> Subject: Re: [Xilinx/Vitis-AI] Missing model annotations in TF2 quantizer (Issue #628) This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model? It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] This should be another issue, but 1) what version of Vitis AI and 2) what's the code being used to optimize the model? It also gave me this issue when I do quantizer = vitis_quantize.VitisQuantizer(model) /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] That is just a warning that TF always generates - it is nothing to do with any issues that you might have. But I'm unable to pinpoint the issue causing this Cannot assign to variable BN-0-0/depthwise_kernel:0 due to variable shape (1, 1, 64, 1) and value shape (3, 3, 64, 1) are incompatible —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.> This is the model deep_rx.zip

[foolmarks]

OK, what I see in Netron is that you have 3 separate inputs:

1. RX-data-In with shape [batch, 14,72,2]
2. TX-Pilot-In with shape [batch, 14,72,2]
3. Raw-Channel-Est-In with shape [batch, 14,72,2]

..these inputs are concatenated inside the model, so your quantizer script will need to provide batches of 3 separate data samples with the appropriate shapes. Is that what you are doing?

Just as an aside, I think the structure of your blocks may cause some issues with the Vitis-AI compiler - I think it will be better to have Conv-BN-relu rather than BN-relu-conv.

[AfifaIshtiaq]

OK, what I see in Netron is that you have 3 separate inputs:

1. RX-data-In with shape [batch, 14,72,2]
2. TX-Pilot-In with shape [batch, 14,72,2]
3. Raw-Channel-Est-In with shape [batch, 14,72,2]

..these inputs are concatenated inside the model, so your quantizer script will need to provide batches of 3 separate data samples with the appropriate shapes. Is that what you are doing?

Just as an aside, I think the structure of your blocks may cause some issues with the Vitis-AI compiler - I think it will be better to have Conv-BN-relu rather than BN-relu-conv.

I use this script to input quantizer Preprocess.zip

[AfifaIshtiaq]

/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: []

when I run results = model.evaluate(x=test_generator) it works fine but when I use vitis quantizer it gets stucand after 15 to 20 mins it gave me the following output

[VAI INFO] Update custom_layer_type: [] Traceback (most recent call last): File "test_model.py", line 50, in

quantized_model = quantizer.quantize_model(calib_dataset=test_generator,separate_conv_act=False)

File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 566, in quantize_model self.optimize_model() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 454, in optimize_model self._create_optimized_model() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 228, in _create_optimized_model quantize_strategy=self._quantize_strategy) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_quantize.py", line 700, in create_optimize_model model, candidate_layers, layer_metadata) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_transforms_pipeline.py", line 74, in apply model, configs, available_transforms, candidate_layers, layer_metadata) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/eight_bit/vitis_8bit_transforms_pipeline.py", line 50, in _apply_availables layer_metadata).recursive_transform() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 738, in recursive_transform self.layer_metadata).transform() File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 704, in transform self._set_layer_weights(layer, weights_map) File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/tensorflow_model_optimization/python/core/quantization/keras/vitis/graph_transformations/model_transformer.py", line 593, in _set_layer_weights K.batch_set_value(weight_value_tuples) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper return target(*args, **kwargs) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow/python/keras/backend.py", line 3576, in batch_set_value x.assign(np.asarray(value, dtype=dtype(x))) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py", line 858, in assign self._shape.assert_is_compatible_with(value_tensor.shape) File "/home/vitis-ai-user/.local/lib/python3.7/site-packages/tensorflow/python/framework/tensor_shape.py", line 1134, in assert_is_compatible_with raise ValueError("Shapes %s and %s are incompatible" % (self, other)) ValueError: Shapes (64,) and (128,) are incompatible

[AfifaIshtiaq]

OK, what I see in Netron is that you have 3 separate inputs:

1. RX-data-In with shape [batch, 14,72,2]
2. TX-Pilot-In with shape [batch, 14,72,2]
3. Raw-Channel-Est-In with shape [batch, 14,72,2]

..these inputs are concatenated inside the model, so your quantizer script will need to provide batches of 3 separate data samples with the appropriate shapes. Is that what you are doing?

Just as an aside, I think the structure of your blocks may cause some issues with the Vitis-AI compiler - I think it will be better to have Conv-BN-relu rather than BN-relu-conv.

I was able to quantize. Is it normal to have the following compile related warning

---

/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] [VAI INFO] Start CrossLayerEqualization... 10/10 [==============================] - 11s 1s/step [VAI INFO] CrossLayerEqualization Done. [VAI INFO] Start Quantize Calibration... 2022-05-17 02:41:53.466988: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) 2701/7200 [==========>...................] - ETA: 3:31:23 7200/7200 [==============================] - 20320s 3s/step [VAI INFO] Quantize Calibration Done. [VAI INFO] Start Post-Quantize Adjustment... [VAI INFO] Post-Quantize Adjustment Done. [VAI INFO] Quantization Finished. WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. model.compile_metrics will be empty until you train or evaluate the model.

[foolmarks]

That warning comes from Tensorflow,  not VitisAI.Sent from my Galaxy -------- Original message --------From: AfifaIshtiaq @.> Date: 17/05/2022 16:34 (GMT+01:00) To: Xilinx/Vitis-AI @.> Cc: Mark Harvey @.>, Comment @.> Subject: Re: [Xilinx/Vitis-AI] Missing model annotations in TF2 quantizer (Issue #628)

OK, what I see in Netron is that you have 3 separate inputs:

RX-data-In with shape [batch, 14,72,2] TX-Pilot-In with shape [batch, 14,72,2] Raw-Channel-Est-In with shape [batch, 14,72,2]

..these inputs are concatenated inside the model, so your quantizer script will need to provide batches of 3 separate data samples with the appropriate shapes. Is that what you are doing? Just as an aside, I think the structure of your blocks may cause some issues with the Vitis-AI compiler - I think it will be better to have Conv-BN-relu rather than BN-relu-conv.

I was able to quantize. Is it normal to have the following compile related warning

/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] [VAI INFO] Start CrossLayerEqualization... 10/10 [==============================] - 11s 1s/step [VAI INFO] CrossLayerEqualization Done. [VAI INFO] Start Quantize Calibration... 2022-05-17 02:41:53.466988: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) 2701/7200 [==========>...................] - ETA: 3:31:23 7200/7200 [==============================] - 20320s 3s/step [VAI INFO] Quantize Calibration Done. [VAI INFO] Start Post-Quantize Adjustment... [VAI INFO] Post-Quantize Adjustment Done. [VAI INFO] Quantization Finished. WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. model.compile_metrics will be empty until you train or evaluate the model.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[AfifaIshtiaq]

That warning comes from Tensorflow,  not VitisAI.Sent from my Galaxy -------- Original message --------From: AfifaIshtiaq @.> Date: 17/05/2022 16:34 (GMT+01:00) To: Xilinx/Vitis-AI @.> Cc: Mark Harvey @.>, Comment @.> Subject: Re: [Xilinx/Vitis-AI] Missing model annotations in TF2 quantizer (Issue #628) OK, what I see in Netron is that you have 3 separate inputs: RX-data-In with shape [batch, 14,72,2] TX-Pilot-In with shape [batch, 14,72,2] Raw-Channel-Est-In with shape [batch, 14,72,2] ..these inputs are concatenated inside the model, so your quantizer script will need to provide batches of 3 separate data samples with the appropriate shapes. Is that what you are doing? Just as an aside, I think the structure of your blocks may cause some issues with the Vitis-AI compiler - I think it will be better to have Conv-BN-relu rather than BN-relu-conv. I was able to quantize. Is it normal to have the following compile related warning /opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.7/site-packages/keras/utils/generic_utils.py:497: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument. category=CustomMaskWarning) [VAI INFO] Update custom_layer_type: [] [VAI INFO] Start CrossLayerEqualization... 10/10 [==============================] - 11s 1s/step [VAI INFO] CrossLayerEqualization Done. [VAI INFO] Start Quantize Calibration... 2022-05-17 02:41:53.466988: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) 2701/7200 [==========>...................] - ETA: 3:31:23 7200/7200 [==============================] - 20320s 3s/step [VAI INFO] Quantize Calibration Done. [VAI INFO] Start Post-Quantize Adjustment... [VAI INFO] Post-Quantize Adjustment Done. [VAI INFO] Quantization Finished. WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. model.compile_metrics will be empty until you train or evaluate the model. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

Thanks.. I'm able to compile my model and now I try to run xdputil on my model but it ran into an error. Although I can run it for other examples

(vitis-ai-tensorflow2) Vitis-AI /workspace/models/ChEstModel > vai_c_tensorflow2 -m quantized_model.h5 -a /opt/vitis_ai/compiler/arch/DPUCVDX8G/VCK190/VCK190arch.json -o compile -n deeprx_compile

---

• VITIS_AI Compilation - Xilinx Inc.

  ---

  [INFO] Namespace(batchsize=1, inputs_shape=None, layout='NHWC', model_files=['quantized_model.h5'], model_type='tensorflow2', named_inputs_shape=None, out_filename='/tmp/deeprx_compile_org.xmodel', proto=None) [INFO] tensorflow2 model: /workspace/models/ChEstModel/quantized_model.h5 [INFO] keras version: 2.6.0 [INFO] Tensorflow Keras model type: functional [INFO] parse raw model :100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 103/103 [00:00<00:00, 18796.26it/s]
  [INFO] infer shape (NHWC) :100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 176/176 [00:00<00:00, 2965.26it/s]
  [INFO] perform level-0 opt :100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 66.53it/s]
  [INFO] perform level-1 opt :100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 280.69it/s]
  [INFO] generate xmodel :100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 176/176 [00:00<00:00, 3567.46it/s]
  [INFO] dump xmodel: /tmp/deeprx_compile_org.xmodel [UNILOG][INFO] Compile mode: dpu [UNILOG][INFO] Debug mode: function [UNILOG][INFO] Target architecture: DPUCVDX8G_ISA2_C64B5_0602001036088252 [UNILOG][INFO] Graph name: DeepRx, with op num: 420 [UNILOG][INFO] Begin to compile... [UNILOG][INFO] Total device subgraph number 33, DPU subgraph number 15 [UNILOG][INFO] Compile done. [UNILOG][INFO] The meta json is saved to "/workspace/models/ChEstModel/compile/meta.json" [UNILOG][INFO] The compiled xmodel is saved to "/workspace/models/ChEstModel/compile/deeprx_compile.xmodel" [UNILOG][INFO] The compiled xmodel's md5sum is c7bf4ec4c241d11c58bc3f47e1122829, and has been saved to "/workspace/models/ChEstModel/compile/md5sum.txt"

Target

root@xilinx-vck190-2021_2:~/models# xdputil benchmark resnet50.xmodel -i -1 1 XAIEFAL: INFO: Resource group Avail is created. XAIEFAL: INFO: Resource group Static is created. XAIEFAL: INFO: Resource group Generic is created. WARNING: Logging before InitGoogleLogging() is written to STDERR I0517 08:39:34.758364 1172 test_dpu_runner_mt.cpp:475] shuffle results for batch... I0517 08:39:34.759827 1172 performance_test.hpp:73] 0% ... I0517 08:39:40.759963 1172 performance_test.hpp:76] 10% ... I0517 08:39:46.760113 1172 performance_test.hpp:76] 20% ... I0517 08:39:52.760265 1172 performance_test.hpp:76] 30% ... I0517 08:39:58.760419 1172 performance_test.hpp:76] 40% ... I0517 08:40:04.760569 1172 performance_test.hpp:76] 50% ... I0517 08:40:10.760716 1172 performance_test.hpp:76] 60% ... I0517 08:40:16.760866 1172 performance_test.hpp:76] 70% ... I0517 08:40:22.761023 1172 performance_test.hpp:76] 80% ... I0517 08:40:28.761174 1172 performance_test.hpp:76] 90% ... I0517 08:40:34.761327 1172 performance_test.hpp:76] 100% ... I0517 08:40:34.761395 1172 performance_test.hpp:79] stop and waiting for all threads terminated.... I0517 08:40:34.763039 1172 performance_test.hpp:85] thread-0 processes 142620 frames I0517 08:40:34.763065 1172 performance_test.hpp:93] it takes 1652 us for shutdown I0517 08:40:34.763082 1172 performance_test.hpp:94] FPS= 2376.87 number_of_frames= 142620 time= 60.0033 seconds. I0517 08:40:34.763118 1172 performance_test.hpp:96] BYEBYE Test PASS. root@xilinx-vck190-2021_2:~/models# xdputil benchmark deeprx_compile.xmodel -i -1 1 XAIEFAL: INFO: Resource group Avail is created. XAIEFAL: INFO: Resource group Static is created. XAIEFAL: INFO: Resource group Generic is created. WARNING: Logging before InitGoogleLogging() is written to STDERR F0517 08:40:51.740269 1177 fix.cpp:32] Check failed: round_mode == "DPU_ROUND" (PY3_ROUND vs. DPU_ROUND) TODO: ** Check failure stack trace: /usr/bin/xdputil: line 20: 1176 Aborted /usr/bin/python3 -m xdputil $***

oot@xilinx-vck190-2021_2:~/models# xdputil benchmark deeprx_compile.xmodel 1 WARNING: Logging before InitGoogleLogging() is written to STDERR F0518 01:43:47.955123 1277 dpu_runner.cpp:206] [UNILOG][FATAL][VART_RUNNER_CONSTRUCTION_FAIL][Cannot create runner] Cannot find runner attr, this subgraph may not be compiled! subgraph name: subgraph_Raw-Channel-Est-In Check failure stack trace: /usr/bin/xdputil: line 20: 1276 Aborted /usr/bin/python3 -m xdputil $*

[deremustapha]

@AfifaIshtiaq How were you able to resolve the Value Error problem. I am kind of struck with the same problem as yours. Thanks a lot.

[AfifaIshtiaq]

Please check if your layer parameters are supported by cpu task and dpu task. Also of dense layer is included in the supported set of operations of vitis ai quantise. Please refer to User guide of vitis ai quantise On Thu 23. Jun 2022 at 08:41, Dere Mustapha Deji @.***> wrote:

@AfifaIshtiaq https://github.com/AfifaIshtiaq How were you able to resolve the Value Error problem. I am kind of struck with the sample problem as yours. Thanks a lot. [image: Error] https://user-images.githubusercontent.com/46024162/175232354-b0e02e14-25b5-471f-bfdb-ae782b70ae40.png [image: H5] https://user-images.githubusercontent.com/46024162/175232350-cb079072-70d4-41d3-8f66-9b694f29a0e0.png

— Reply to this email directly, view it on GitHub https://github.com/Xilinx/Vitis-AI/issues/628#issuecomment-1164013833, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQIQY4VA26E5R7QMM44P3WTVQQBLHANCNFSM5LXTCWGA . You are receiving this because you were mentioned.Message ID: @.***>

[AfifaIshtiaq]

Also dropout layers can be an issue

On Thu 23. Jun 2022 at 08:41, Dere Mustapha Deji @.***> wrote:

@AfifaIshtiaq https://github.com/AfifaIshtiaq How were you able to resolve the Value Error problem. I am kind of struck with the sample problem as yours. Thanks a lot. [image: Error] https://user-images.githubusercontent.com/46024162/175232354-b0e02e14-25b5-471f-bfdb-ae782b70ae40.png [image: H5] https://user-images.githubusercontent.com/46024162/175232350-cb079072-70d4-41d3-8f66-9b694f29a0e0.png

— Reply to this email directly, view it on GitHub https://github.com/Xilinx/Vitis-AI/issues/628#issuecomment-1164013833, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQIQY4VA26E5R7QMM44P3WTVQQBLHANCNFSM5LXTCWGA . You are receiving this because you were mentioned.Message ID: @.***>