search

firebase / firebase-ios-sdk

Firebase SDK for Apple App Development
https://firebase.google.com
Apache License 2.0
5.67k stars 1.49k forks source link

[ML Text Recognition] Different result on different iOS device #5575

Closed artemsivcev closed 4 years ago

artemsivcev commented 4 years ago

[REQUIRED] Step 1: Describe your environment

[REQUIRED] Step 2: Describe the problem

Steps to reproduce:

Hey there! I have 2 same images and 2 iPhones: 8 and X. UIImage have scale set to 1 and cropped width and height by app. Then I send this image in onDeviceTextRecognizer() and get 2 different maps with all different results. Why so? Is there some scale dependence in ML KIT library? Images is not too small. height = 2048, width = 1383. I have same result without cropping image.

artemsivcev commented 4 years ago

For tests, I compressed the image to 1280px, but still have different result, for different iPhones...

artemsivcev commented 4 years ago
Now I took image from example and there is still different result for iPhone 8 and X. Prop iPhone 8 iPhone X
Height 1536.0 1536.0
Width 1024.0 1024.0
Scale 1.0 1.0
Wege der [(410, 666) - (631, 643)][(414, 710) - (636, 688)] [(410, 667) - (631, 644)][(414, 713) - (636, 691)]
parlamentarischen [(287, 738) - (743, 686)][(292, 785) - (748, 733)] [(287, 738) - (742, 685)][(292, 785) - (747, 733)]
Demokratie [(372, 786) - (651, 754)][(376, 825) - (655, 794)] [(372, 785) - (650, 754)][(376, 825) - (654, 794)]

As you can see there is some mismatch for text coordinates for same image on different devices. And if you try to proceed this image several times you have got same result per device. But why and how to avoid this?

artemsivcev commented 4 years ago

Sorry missclick...

paulb777 commented 4 years ago

@artemsivcev Thanks for the report. We're tracking internally at b/156483920

artemsivcev commented 4 years ago

@paulb777 is there any chance for hotfix to this issue?

artemsivcev commented 4 years ago
Now I tested version 0.12.0 and 0.18.0 for this bug and there it is, but with different coordinates. In ver.12 there is more accuracy between devices: Prop iPhone 8 iPhone X
Height 1536.0 1536.0
Width 1024.0 1024.0
sclae 1.0 1.0
Wege der [(412, 665) - (633, 645)][(416, 713) - (637, 694)] [(410, 665) - (633, 645)][(414, 713) - (637, 694)]
parlamentarischen [(287, 739) - (743, 688)][(292, 784) - (748, 734)] [(287, 739) - (743, 688)][(292, 784) - (748, 734)]
Demokratie [(372, 785) - (650, 754)][(376, 824) - (654, 794)] [(372, 785) - (651, 754)][(376, 824) - (655, 794)]

And in ver.18 same result as ver.19

artemsivcev commented 4 years ago

@paulb777 Hi! Any progress?

paulb777 commented 4 years ago

@artemsivcev Sorry, not yet. I repinged the MLKit team.

artemsivcev commented 4 years ago

@paulb777 sorry for trying, but... Still nothing? We are used this solution on production and this is really major issue...

artemsivcev commented 4 years ago

@paulb777 Hi! Any progress?) I can help if you need it.

artemsivcev commented 4 years ago

@morganchen12 ok, I can ping you instead)) I'm really want to resolve this issue and help!

morganchen12 commented 4 years ago

Hey, sorry for the delay @artemsivcev. I'll take a look at this today. Are you receiving different results on different devices if you use the exact same image (not just two separate images downscaled to the same size)?

artemsivcev commented 4 years ago

@morganchen12 yes, I sent on 2 iPhones 1 image via AirDrop. Then I open it with UIImage and compare all meta info and it was exactly the same. Where can I check downscaled info?

morganchen12 commented 4 years ago

Your approach should be sufficient. I'll investigate.

morganchen12 commented 4 years ago

I haven't been able to identify a root cause yet. I'll try reproducing this issue in the quickstart and continue from there.

artemsivcev commented 4 years ago

Ok, if you need any info or help let me know

artemsivcev commented 4 years ago

@morganchen12 Some update, on all simulators same result for coordinates, problem only on real devices

morganchen12 commented 4 years ago

Thanks @artemsivcev, I was also unable to reproduce the issue on simulator. Due to the work from home situation, I'm not immediately able to acquire another iOS device to test this on, so I'm going to have to ask for your help for on-device testing.

ML Kit's text recognition uses GoogleMobileVision under the hood. Can you try running text detection directly through GMV via the instructions in this doc and letting us know if the results differ per-device?

artemsivcev commented 4 years ago

@morganchen12 yeah of course, I'll try it right now

artemsivcev commented 4 years ago
Same test image, same test devices. Same different result. But please, don't redirect me to GoogleMobileVision repo) Prop iPhone 8 iPhone X
Height 1536.0 1536.0
Width 1024.0 1024.0
scale 1.0 1.0
Wege der [410.0, 644.0, 229.0, 68.0] [410.0, 645.0, 229.0, 67.0]
parlamentarischen\nDemokratie [287.0, 688.0, 467.0, 147.0] [288.0, 688.0, 465.0, 146.0]
morganchen12 commented 4 years ago

Thanks, I'll pass this info on to the GMV team. In the meantime, what version of GMV are you using? Does the behavior change if you switch to an older version?

artemsivcev commented 4 years ago

@morganchen12 I get GMV from CocoaPods, think that was last one. Without issue in GMV repo(there is one?) it's dead end... Whom I suppose to ping? =D Need a solution for this...

morganchen12 commented 4 years ago

You can manually pin an older version of a CocoaPod by explicitly listing it in your Podfile with the = operator. Don't worry about pinging the GMV team, I've done that for you (their issue tracker is Google-internal).

pod 'GoogleMobileVision', `= 1.5.0`
artemsivcev commented 4 years ago

Only 1.5.0 need to check? And how to see all a available version on pods?

morganchen12 commented 4 years ago

You should hopefully only need to check 1.5.0. You can check which versions of a pod are available by looking at the podspecs on GitHub: https://github.com/CocoaPods/Specs/tree/master/Specs/0/4/2/GoogleMobileVision

artemsivcev commented 4 years ago

@morganchen12 GMV 1.5.0 test. There is only one line for iPhoneX... Still different result, but something new!

Prop iPhone 8 iPhone X
Height 1536.0 1536.0
Width 1024.0 1024.0
scale 1.0 1.0
Wege der [414.0, 643.0, 233.0, 69.0] [-]
parlamentarischen\nDemokratie [290.0, 687.0, 468.0, 155.0] [-]
Wege der parlamentarischen Demokratie [-] [284.0, 635.0, 471.0, 205.0]
morganchen12 commented 4 years ago

Thanks, looks like this behavior has been broken for some time now. I'll keep this updated with the GMV team's response, which may take a week or two. In the meantime let's try to find a workaround for your use case.

What are you using text recognition for and why is it important that text recognition is consistent across different devices?

artemsivcev commented 4 years ago

I'm using it for detect table data. Some photo contains problems with degree angle for text. I try to use angle of bottom line of each block to detect average angle and create a straight line between blocks. And I create the best solution and it works, but only on iPhone 8. Now I found some workaround and used only longest blocks for detect angle. It's pretty close, but still have some issues...

artemsivcev commented 4 years ago

@morganchen12 hey there! Any progress?

morganchen12 commented 4 years ago

I have not been able to pin down why GMV returns different results on different devices. I'll let you know if I'm able to find a root cause.

artemsivcev commented 4 years ago

@morganchen12 Hello! Still nothing? GMV team keep silent?

morganchen12 commented 4 years ago

Nothing yet. I've pinged the internal bug for you.

artemsivcev commented 4 years ago

That would be nice) And what they say?

morganchen12 commented 4 years ago

They have yet to respond.

morganchen12 commented 4 years ago

The Firebase ML Kit text recognition API has been deprecated since all on-device ML Kit features are no longer a part of Firebase. You should move to using the Google standalone ML Kit instead by following the migration guide here.

Sorry for the bad news.