search

john-rocky / CoreML-Models

Converted CoreML Model Zoo.
1.32k stars 121 forks source link

Performance and Segmentation Accuracy Issues with SegFormer CoreML Model #34

Open rouzbeh-abadi opened 1 year ago

rouzbeh-abadi commented 1 year ago

Hello,

I'm integrating the SegFormer CoreML model into an iOS project for real-time semantic segmentation but I'm facing two major issues:

Performance: The model is slower than expected in real-time settings, which is considerably affecting its usability.

Segmentation Accuracy: The model's outputs seem incorrect in terms of class predictions and their spatial locations. The segmentations do not align with the actual positions of the objects in the input image.

Additionally, while initializing the model, I receive a warning in Xcode: "Sort/TopK layer does not support input tensor's shape along vector dimension > 8." I'm wondering if this could be contributing to the issues with the segmentation output.

Any guidance on addressing these issues would be highly appreciated.

Thank you in advance for your assistance.

john-rocky commented 8 months ago

Make sure the input image is cropped correctly. If you are using VNCoreMLRequest, make sure imageCropAndScaleOption is set to scallFill.