search

googlesamples / mlkit

A collection of sample apps to demonstrate how to use Google's ML Kit APIs on Android and iOS
Apache License 2.0
3.51k stars 2.92k forks source link

Face Detection on iOS: empty results for front camera [with screenshots] when image is not scaled using extension utility #406

Closed ozpopolam closed 2 years ago

ozpopolam commented 2 years ago

Hi! I'm reopening this issue (https://github.com/googlesamples/mlkit/issues/404) with some screenshots.

I've just tried to follow your exact steps again by cloning samples and running 'pod update' for mlkit/ios/quickstarts/vision/Podfile. I've also tried to run demo on my other devices (iPhone 6 with 12.5.4 and iPhone XS with 14.8.1) and I always get the same result: 1) front camera with your initial code: CK3i1Iw

2) front camera with replaced scaledImage(with:)- method to returning self: xTnkWgC

3) rear camera with replaced scaledImage(with:)- method: JkdzkIN

Looks like face features are drawn rotated (I think it's because scaling method also rotates image to .up orientation, and as this step is skipped, following transformation of the face.frame is not correct), but my face has been detected.

If it helps, I use Xcode 13.0 and CocoaPods 1.11.2.

Maybe I could send you some original photos so you'll try to reproduce on them?

miworking3 commented 2 years ago

I see, you are using the camera photo, not camera video. Now I can reproduce it, will take a look.

miworking3 commented 2 years ago

After hacking the quickstart app by replacing these lines with return self, it will get wrong orientation of the photo: the photo was taken portrait, apparently, the size tells that the orientation can't be right:

That's why the face is not detected.

Specifying the correct orientation is a pre-requirement for the face detection, if you don't want to use the utilities in our sample app, you need to make sure that your orientation is set correctly.

ozpopolam commented 2 years ago

@miworking3, thanks for reproducing!

I'm still confused, though: all portraits (whether they have been taken with front or rear cameras) returns .right as their orientation, which corresponds to the value 6 stored in EXIF. For front camera (and iPhone XS) it looks like: (2316.0, 3088.0) + .right orientation For rear camera: (3024.0, 4032.0) + .right orientation.

And this orientation is set in demo here in a same way both for front and rear portraits, but detection works for rear one only.

I've also tried to manually set visionImage.orientation for front portraits to all possible 8 orientations (to check which one will be suitable), but still no successful detection.