vloncar
commented
1 year ago Hi Lorenzo,
This is really interesting, we would definitely welcome an implementation of instance normalization based on the work you already started. From your description, the only thing you're missing is the HLS/C++ code that contains the implementation (maybe also the config/function call templates). You can look at the examples of extension API for what is needed to support a new layer, for example here and here. In your case this will be the core new feature, so it won't need calls to register_* functions and will be implemented the way you started it.
Prerequisites
I have spoken to the hls4ml team on the GitHub discussions page for ideas and confirmed that the instance normalization layer is not currently in active development.
Details
Adding the InstanceNormalization Layer from the TensorFlow-Addons package. What I have done so far is add to keras_handler_file a modified function for the InstanceNormalization, and make the registration, plus I added in the model.layers a class Instancernormalization. I can arrive a the hls_model.write() and I can create myproject.so, should I also modify/add something to the cpp code created?
New behaviour
I would like to request that the instance normalization layer be added to hls4ml. The instance normalization layer is commonly used in deep learning architectures for image processing and is currently missing from the hls4ml library. I am specifically interested in using this layer for anomaly detection in High Energy Physics (HEP) data and am using it as part of a Joint Variational Autoencoder (VAE) model.
Motivation
The addition of the instance normalization layer would be very valuable to the hls4ml community. This layer is commonly used in computer vision tasks such as image segmentation and object detection and would enable users to convert these types of networks to hardware more easily. Additionally, instance normalization has been shown to improve the performance of deep neural networks in a variety of tasks, so it would be a valuable addition to the hls4ml library.
In the context of HEP, converting the instance normalization layers to hardware using hls4ml would be especially useful for anomaly detection and other classification tasks. Adding this layer to hls4ml would enable researchers in HEP and related fields to experiment more freely and implement these models on hardware.
Parts of hls4ml being affected
The addition of the instance normalization layer would affect the core functionality of hls4ml, as it would require modifications to the existing layers and backends to support the new layer. It may also require updates to the existing interface and configuration options in order to support the new layer properly.