🐛 [iOS] Frame processor doesn't work correctly in portrait mode

[alarm109]

What were you trying to do?

I am passing frame to frame processor plugin which uses MLKit. I am trying to label fruits. When the app is in landscape mode it works correctly. But when the app is in portrait mode, I get random labelling when pointing to banana, e.g. "solar dish", "radiator".

After a little debugging. I can see that frame always gives orientation "right" in any position. So maybe the frame is created incorrectly somehow when in portrait mode?

p.s. I'm using local model from tensorflow, but the same issue happens with default MLKit labeling.

Reproduceable Code

@objc
  public static func callback(_ frame: Frame!, withArgs args: [Any]!) -> Any! {
    guard let buffer = frame.buffer else {
      return nil
    }
    let orientation = frame.orientation

    let image = VisionImage(buffer: buffer)
    image.orientation = orientation

    let fileName = "mobilenet_v1_1.0_224_quantized_1_metadata_1"
    let fileExtension = "tflite"
    guard let path = Bundle.main.path(forResource: fileName, ofType: fileExtension) else {
      NSLog("No model file.")
      return nil
    }

    let localModel = LocalModel(path: path)
    let options = CustomImageLabelerOptions(localModel: localModel)

    options.confidenceThreshold = 0.3
    let labeler = ImageLabeler.imageLabeler(options: options)

    do {
      let labels = try labeler.results(in: image)
      var results: [ Any ] = []

      for label in labels {
        var result: [ String:Any ] = [:]
        result["label"] = label.text
        result["confidence"] = label.confidence
        result["index"] = label.index
        results.append(result)
      }

      return results
    } catch {
      NSLog("Failed to get label.")
    }

    return nil
  }

What happened instead?

Frame is incorrect, therefore labelling in portrait mode is incorrect.

Relevant log output

No response

Device

iPhone 12 (iOS 15.3.1)

VisionCamera Version

2.12.2

Additional information

• [ ] I am using Expo
• [X] I have read the Troubleshooting Guide
• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar issues in this repository and found none.

[mrousavy]

Hey! Thanks for the report, are you saying the orientation prop never changes? Not even if your app rotates?

[alarm109]

Yes, I changed in deployment info to just Portrait or just Landscape to check. The frame's orientation stayed the same. I always seem to get orientation value "right".

If this is always the case, my guess would be that maybe the frame is generated in the incorrect orientation. Therefore even if I put the correct orientation to MLKit it still gives back incorrect answer, because the buffer is generated incorrectly? Could be completely wrong, because I don't really know how the frames are generated in this case

[andresprimera]

Hi there! Is this fixed? Because it is happening to me as well using mlkit for object detection. It works even upside down, but not portrait mode.

[mrousavy]

Hey - is this still an issue? I think orientation is now just always portrait, I will revisit the Orientation issue once I get enough sponsors for that as it is quite compelx to solve it once and for all

[mrousavy]

Hey - I'm tracking Orientation in this feature request/issue now: https://github.com/mrousavy/react-native-vision-camera/issues/1891 ✨

Make sure to upvote or sponsor to support this feature, and leave a comment if you have any additional thoughts/ideas.