Channels Last Format i.e. How to build (N, H, W, C) Models

[dino-illumix]

Is there any way to create an mmsegmentation model with an Input and Output format (N, H, W. C) ? ie with the channels as the last dimension of the input and output? Ultimately I want to convert an mmsegmentation model to ONNX and then TFLite format for on-device implementation. I need to have the channels as the last dimension (also this may be more efficient) performance wise) but there doesn't seem to be any way to create a model with this ordering of the dimensions. I've tried setting in_channels=(1, 512, 512, 3) and out_channels=(1, 512, 512, 1) in the Backbone config and setting data_format='channels_last' but neither of these seem to make any difference and the resulting model is always in (N, C, H, W) format.

[jonGuti13]

Hi @dino-illumix. How have you converted your mmsegmentation model to ONNX? Thanks in advance.

[dino-illumix]

Yes. I've been able to convert to successfully convert to ONNX using mmdeploy tool and subsequently to TfLite, however there is no functionality in the ONNX conversion process that allows conversion of NCHW mmsegmentation models to NHWC format

[jonGuti13]

Thank you very much. I have been able to convert standard mmsegmentation models to ONNX. However, the conversion of some custom models is not straightforward and I'm stuck there. When I solve that problem, I will have a look at the (N, C, H, W) / (N, H, W, C) issue.

[dino-illumix]

Many thanks. I had a look at some of the underlying mmcv code and it all seems to be geared towards NCHW. I am building for mobile devices though and NHWC is likely to be better for performance reasons, as well as that the code I am integrating into expects NHWC, so it would be very helpful to be able to convert format when creating the ONNX, or else at training time in mmsegmentation configuration.

[jonGuti13]

My approach is completely different. I want to use some SOTA models to train them with my own database and compare those results with the ones from my models.

By the way, could you please tell me how you are converting your models from ONNX to TFLite? Are you using onnx-tensorfow and the following commands?

import onnx
from onnx_tf.backend import prepare

model = onnx.load(onnx_path)
tf_rep = prepare(model)
tf_rep.export_graph(tf_model_path)

This way, I have been able to successfully export to Tensorflow SavedModel an mmdetection ONNX file which was created like this:

python ./tools/deploy.py ./configs/mmdet/detection/detection_onnxruntime_dynamic.py
        mmdetection/configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py \
        checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
        mmdetection/demo/demo.jpg --work-dir mmdeploy_model/faster-rcnn \
        --device cpu --dump-info

However, following the same approach for the following two mmsegmentation models...

python ./tools/deploy.py ./configs/mmseg/segmentation_onnxruntime_dynamic.py \
    mmsegmentation/configs/unet/unet-s5-d16_fcn_4xb4-160k_cityscapes-512x1024.py \
    mmsegmentation/checkpoints/fcn_unet_s5-d16_4x4_512x1024_160k_cityscapes_20211210_145204-6860854e.pth \
    mmsegmentation/demo/demo.png \
    --work-dir mmdeploy_model/fcn_unet \
    --device cpu \
    --dump-info

python ./tools/deploy.py ./configs/mmseg/segmentation_onnxruntime_dynamic.py \
    mmsegmentation/configs/fcn/fcn_r50-d8_4xb2-40k_cityscapes-512x1024.py \
    mmsegmentation/checkpoints/fcn_r50-d8_512x1024_40k_cityscapes_20200604_192608-efe53f0d.pth \
    mmsegmentation/demo/demo.png \
    --work-dir mmdeploy_model/fcn \
    --device cpu \
    --dump-info

I haven't been able to export them to TF as I get this strange error

File "/workspace/Vitis-AI/tutorials/onnx-tensorflow/onnx_tf/backend_tf_module.py", line 99, in __call__  *
    output_ops = self.backend._onnx_node_to_tensorflow_op(onnx_node,
File "/workspace/Vitis-AI/tutorials/onnx-tensorflow/onnx_tf/backend.py", line 347, in _onnx_node_to_tensorflow_op  *
    return handler.handle(node, tensor_dict=tensor_dict, strict=strict)
File "/workspace/Vitis-AI/tutorials/onnx-tensorflow/onnx_tf/handlers/handler.py", line 58, in handle  *
    cls.args_check(node, **kwargs)
File "/workspace/Vitis-AI/tutorials/onnx-tensorflow/onnx_tf/handlers/backend/resize.py", line 125, in args_check  *
    exception.OP_UNSUPPORTED_EXCEPT(
File "/workspace/Vitis-AI/tutorials/onnx-tensorflow/onnx_tf/common/exception.py", line 50, in __call__  *
    raise self._func(self.get_message(op, framework))
RuntimeError: Resize coordinate_transformation_mode=pytorch_half_pixel is not supported in Tensorflow.

On the other hand, following a different path, I have also used onnx2keras to directly export an ONNX model to Keras. I imagine you know about it, but just in case, because it could be useful for you. In the past and still now, however, I have faced similar issues to the ones you are reporting (related to NHWC vs NCHW), even though there exists an experimental feature to change the ordering of the channels (https://github.com/gmalivenko/onnx2keras). None of the above-mentioned 3 models have been succesfully converted and the error has been

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.8/site-packages/onnx2keras/converter.py", line 175, in onnx_to_keras
    AVAILABLE_CONVERTERS[node_type](
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.8/site-packages/onnx2keras/convolution_layers.py", line 177, in convert_conv
    layers[node_name] = conv(input_0)
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1751, in _create_c_op
    raise ValueError(e.message)
ValueError: Exception encountered when calling layer "input.3" (type Conv2D).

Negative dimension size caused by subtracting 3 from 2 for '{{node input.3/Conv2D}} = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], explicit_paddings=[], padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true](Placeholder, input.3/Conv2D/ReadVariableOp)' with input shapes: [?,3,2,2], [3,3,3,64].

The ONNX model has been created with ./configs/mmseg/segmentation_onnxruntime_dynamic.py. If I change that configuration file and use a static one such as ./configs/mmseg/segmentation_onnxruntime_static-512x512.py I still get an error but it is almost at the end of the network and it has nothing to do with channel ordering.

DEBUG:onnx2keras:Check input 0 (name onnx::Slice_549).
DEBUG:onnx2keras:Check input 1 (name onnx::Slice_551).
DEBUG:onnx2keras:Check input 2 (name onnx::Slice_552).
DEBUG:onnx2keras:Check input 3 (name onnx::Slice_550).
DEBUG:onnx2keras:... found all, continue
DEBUG:onnx2keras:slice:Slice numpy constants
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.8/site-packages/onnx2keras/converter.py", line 175, in onnx_to_keras
    AVAILABLE_CONVERTERS[node_type](
  File "/opt/vitis_ai/conda/envs/vitis-ai-tensorflow2/lib/python3.8/site-packages/onnx2keras/reshape_layers.py", line 294, in convert_slice
    raise AttributeError('Not implemented')
AttributeError: Not implemented

For completeness, I show the output of netron when the onnx has been created with a dynamic configuration script

and with a static configuration script. The difference is blatantly obvious.

Maybe most of what I've mentioned won't be of much use to you, but let it not be for lack of trying :)

[dino-illumix]

"By the way, could you please tell me how you are converting your models from ONNX to TFLite? Are you using onnx-tensorfow and the following commands?"

Yes, that's exactly how I've been doing it. The same way as you and using deploy.py to create the onnx file in the first place.