Post-processing error adding IoMapEntry for CenterCrop exporting YoloV8 model

[frtendero]

Issue with IoMapEntry for CenterCrop in YOLOv8 Model Export to ONNX

Hey there :)

Description

I am using onnxruntime-extensions to export my YOLOv8 model to ONNX, adding the necessary pre and post-processing operations for the segmentation model neural network. In addition to the standard preprocessing operations, I want to add a CenterCrop to the original image.

I added that before the Resize operation and it's working fine, but obviously, I get the detection coordinates shifted, so I need to add the corresponding post-processing step to convert detection coordinates back to the original ones. For that, I guess that I have to include it as IoMapEntry pointing to the CenterCrop:

ScaleNMSBoundingBoxesAndKeyPoints(name="ScaleBoundingBoxes"),
[
    utils.IoMapEntry(
        "ConvertImageToBGR", producer_idx=0, consumer_idx=1
    ),
    utils.IoMapEntry("CenterCrop", producer_idx=0, consumer_idx=2),
    utils.IoMapEntry("Resize", producer_idx=0, consumer_idx=3),
    utils.IoMapEntry("LetterBox", producer_idx=0, consumer_idx=4),
],

However, I encounter the following error:

io_map_entries[entry.consumer_idx] = IoMapEntry(producer, entry.producer_idx, entry.consumer_idx)
IndexError: list assignment index out of range

It appears that the io_map_entries list is of size 4 and contains the following elements: ['nms_step_output', 'original_image', 'resized_image', 'letter_boxed_image'].

Expected Behavior

The code should correctly map the CenterCrop operation in both pre and post-processing steps without causing an index out of range error.

Actual Behavior

The code raises an IndexError: list assignment index out of range when adding the IoMapEntry for CenterCrop.

Steps to Reproduce

1. Add CenterCrop operation to the preprocessing steps.
2. Add the corresponding IoMapEntry for CenterCrop in the post-processing steps.
3. Attempt to export the model to ONNX.

Additional Context

• python version: 3.10.14
• onnxruntime-extensions version: 0.10.1
• onnx version: 1.16.0
• YOLOv8 model details: yolov8m-seg trained with custom data

I'm not sure if I'm proceeding good, or if that maybe is not possible.

Please, feel free to ask me, if needed,the full code for exporting, the visualization of the model in netron.app or whatever you need

Thanks for your help!

[wenbingl]

@skottmckay

[skottmckay]

The implementation only supports 3 specific changes to the image currently, as each change has different processing (scale co-ords for resize, pad co-ords for letterboxing). Adding a 4th value means a lot more complexity around handling it being optionally provided.

Quickest may be to do this outside of the model. I assume it should be relatively easy to calculate the change from a centered crop vs the original image. If you provide the cropped image to ScaleNMSBoundingBoxesAndKeyPoints instead of the original image the co-ords will be relative to that, and you just need to offset the co-ords by half of the cropped amount for the height and width that the centered crop did.

ScaleNMSBoundingBoxesAndKeyPoints(name="ScaleBoundingBoxes"),
[
    utils.IoMapEntry("CenterCrop", producer_idx=0, consumer_idx=1),
    utils.IoMapEntry("Resize", producer_idx=0, consumer_idx=2),
    utils.IoMapEntry("LetterBox", producer_idx=0, consumer_idx=3),
],

e,g, original image was 500 x 800, centered crop to 400 x 600. as we cropped 100 off the height and 200 off the width, the co-ords need to have height +50 and width +100.