[Feature Request] Unsupported Layer for tensorflow addons(tfa) "tfa.layers.GroupNormalization"

[zackdilan]

System information

• TensorFlow version (you are using): TF 2.3.0

• Are you willing to contribute it (Yes/No): Yes

Motivation

1. I think there are many users who want to accelerate custom models which may or may not contain layers that are currently supported by Vitis ai tF2.x quantizer.
2. So I would like to point out certain questions which may some of our other users might have.
3. The Vitis ai documentation misses certain information regarding the TF2.x quantization training which can be also addressed with this issue.
4. I would like to contribute an example, which can show how these can be done. For eg., we may skip the particular layer or create a custom quantization config, etc.
5. Refer to the Misc section for further questions this issue can answer

Describe the feature: Group Normalization divides the channels into groups and computes within each group the mean and variance for normalization.

Describe how the feature helps achieve the use case. Empirically, its accuracy is more stable than the batch norm in a wide range of small-batch sizes, if the learning rate is adjusted linearly with batch sizes.

Describe how existing APIs don't satisfy your use case I am attaching the procedure and TF model and error screenshots:

1. Construct the model
2. Loaded the trained model for quantization as explained in the user guide.
3. [Expected behaviour] unknow layer error

Misc.

1. Are we using post-training quantization or Quantization aware training
2. With the API call that I have used, am I using the Quantization aware training(QAT) quantization method? (refer to step2. screenshot).
3. With QAT, we can skip the particular layer as a turnaround, but what are the next steps? Suppose I trained the above mode with QAT and how to quantize the model then? In TF we have TFLite API s.. what to do here?
4. if we can do the above step, will be the model supported by DPU or can be compiled?

For people who want to contact me for collaboration: robin.chacko@plc2.de

[sheng-xiao]

Hi @zackdilan, The user guide of Vitis AI 1.3 includes some basic information about how to do the PTQ(quantize calibration) and QAT(quantize finetuning) with the tf2 quantizer. 1&2. We support both PTQ and QAT, and in step 2 you are doing PTQ.

3. After the QAT process, you can feed the quantize finetuned model to the compiler (vai_c) to compile and then deploy it on FPGA. Actually the following process is the same as PTQ.
4. Currently, only layers in the support list can be compiled and mapped to run on DPU. Other layers including your customized layer will be mapped to the CPU.
5. The error in snapshot 3 need to register the custom layer before calling quantize_model. Here is an example:

   with keras.utils.CustomObjectScope({'GroupNormalization': GroupNormalization}):
   quantized_model = quantizer.quantize_model(calib_dataset=eval_dataset)

In Vitis AI 1.3 the quantization support for custom layers is experimental, you can do it by register the custom layer in the quantize strategy file (https://github.com/Xilinx/Vitis-AI/blob/master/tools/Vitis-AI-Quantizer/vai_q_tensorflow2.x/tensorflow_model_optimization/python/core/quantization/keras/vitis/vitis_8bit_default_quantize_strategy.json). Please note that this is just for quantization experiments and the compiler cannot handle the custom layers now.

We are improving the interface and ease of use of custom APIs and provide examples and documentation for them. They are on the schedule of the following releases.

[zackdilan]

Hi, @sheng-xiao Thanks a lot for your detailed explanation.

As far as I have obtained,

1. the model: its not a custom layer, but its available in TensorFlow addons(tfa)

2. I can quantize the model with Post-training quantization steps explained in the user guide and along with your example code.(I didn't try custom quantize strategy for now )

3.Finally when I try to compile the model : Expected: layers in the support list are mapped into DPU and other layers mapped to CPU. What I got :

• The error related to Tf to Xmodel creation and there is a no-op that my unsupported layer has. Do we need to make changes in the tf_translator code? Thanks in advance

[zackdilan]

Hi @sheng-xiao , sorry for bothering you again. I would like to have a clear understanding of the unsupported operators.

The Vitis ai compiler only assigns operations to the CPU only if the configuration of the DPU operator exceeds the limitation. FOr example CONV2D operation - dpu supports and its kernel dimension limitation is up to w, h: [1, 16]. So if I have a conv2d layer of 17*17, this will be assigned to CPU right? And consider for a conv2d layer, I use tanh as activation, will it assign to CPU automatically?

I did a small prrof check for tanh activation, but I have realized : And I think the Vitis ai compiler (for now) cannot automatically assign an operation to the CPU for example the Tanh operand(which is not DPU supported). You can find more details in the compile log

compile.log

[renjiangyong]

The GroupNormalization op is not support. This issue is existed for a long time , I will close it ,if you have problem, you can reopen it. Thank you @zackdilan