Method _convertCameraImage not working in IOS?

[doanbh]

I run this app on my Android device then everything works fine. But when running on IOS after I debug, the _convertCameraImage method does not return results. It seems that in that method there is too much computation so it doesn't work on iOS or something like that. Please give me a way to solve this problem. Thank you.

[Rajatkalsotra]

@doanbh This issue is related to the output of camera format (android -> yuv and ios -> bgra) . And the model requires it in bgra format. So just change the function to following code for IOS :

imglib.Image _convertCameraImage(
      CameraImage image, CameraLensDirection _dir) {
imglib.Image img = imglib.Image.fromBytes(
      image.width,
      image.height,
      image.planes[0].bytes,
      format: imglib.Format.bgra,
    );
var img1 = (_dir == CameraLensDirection.front)
        ? imglib.copyRotate(img, -90)
        : imglib.copyRotate(img, 90);
    return img1;
}

[doanbh]

You are right, I found this Android and IOS different problem a few days ago, I found the solution online is to use this function below, I have tried it on Android and iOS both works fine when using use this function together. But there is a problem, when I use Iphone and push array featured recognite and user info login to my server, then I log in that account on another Iphone phone still recognize the face, but login It is not recognized on Android device, although I have only used 1 function to convert images. And conversely, if I use Android to update the featured recognite array at first, then only the Android device will recognize it. Hope you help me, thank you very much. ` imglib.Image _convertCameraImage1(CameraImage image) {

int width = image.width;
int height = image.height;
var img = imglib.Image(width, height);
for(int x=0; x < width; x++) {
  for(int y=0; y < height; y++) {
    final pixelColor = image.planes[0].bytes[y * width + x];
    // color: 0x FF  FF  FF  FF
    //           A   B   G   R
    // Calculate pixel color
    img.data[y * width + x] = (0xFF << 24) | (pixelColor << 16) | (pixelColor << 8) | pixelColor;
  }
}
var img1 = imglib.copyRotate(img, -90);
return img1;

} `

[doanbh]

@doanbh This issue is related to the output of camera format (android -> yuv and ios -> bgra) . And the model requires it in bgra format. So just change the function to following code for IOS :

imglib.Image _convertCameraImage(
      CameraImage image, CameraLensDirection _dir) {
imglib.Image img = imglib.Image.fromBytes(
      image.width,
      image.height,
      image.planes[0].bytes,
      format: imglib.Format.bgra,
    );
var img1 = (_dir == CameraLensDirection.front)
        ? imglib.copyRotate(img, -90)
        : imglib.copyRotate(img, 90);
    return img1;
}

I'm follow this code, it works fine on iOS, but on Android any face is considered the same, and I check the euclideanDistance is very small, only about 0.2, do you have any way to fix this?

[fnoceda]

Hi @Rajatkalsotra thanks for you job, very impresionant.

I have a similar problem here, I save my photo and the array resulting of predictedData on my back end database with a API REST, when I Sign In and Sign Up in a ios-ios or android-android no problem, but when i try authenticate cross platform the distance result different i see the array save on my back end and are different in this case.

IOS SignUp-IOS Sign In [works fine] ANDROID SignUp-ANDROID Sign In [works fine] IOS SignUp-ANDROID Sign In [don't authenticate] ANDROID SignUp-IOS Sign In [don't authenticate]

I log my results IOS SignUp-IOS SignIn // flutter: calculandoDistancia // flutter: currDist=>0.7708154422560634 // flutter: threshold=>1.0 // flutter: minDist=>0.7708154422560634 // flutter: end setPredictedData // flutter: authenticate OK

ANDROID SignUp-ANDROID SignIn // flutter ( 8968): calculandoDistancia // flutter ( 8968): currDist=>0.6422662304939313 // flutter ( 8968): threshold=>1.0 // flutter ( 8968): minDist=>0.6422662304939313 // flutter ( 8968): end setPredictedData // flutter ( 8968): authenticate OK

IOS SignUp-ANDROID SignIn // flutter (19459): calculandoDistancia // flutter (19459): currDist=>1.4399978617184572 // flutter (19459): threshold=>1.0 // flutter (19459): minDist=>999.0 // flutter (19459): end setPredictedData // flutter (19459): don't authenticate

ANDROID SignUp-IOS SignIn // flutter: calculandoDistancia // flutter: currDist=>1.4992889685122506 // flutter: threshold=>1.0 // flutter: minDist=>999.0 // flutter: end setPredictedData // flutter: don't authenticate

The predictedData saved is here (is very diferent in android and ios)

With IOS [-0.005366702564060688, 0.03501761332154274, 0.016875134781003, 0.0008109764894470572, -0.07175101339817047, 0.09405621886253357, -0.05950654298067093, 0.05654291436076164, -0.0009066364145837724, -0.030182218179106712, -0.029889754951000214, 0.0075845494866371155, -0.00888113770633936, 0.02600974030792713, 0.0004703785525634885, 0.00661942083388567, -0.025093356147408485, -0.015354325994849205, -0.00023305673676077276, 0.009817020036280155, -0.19247987866401672, 0.09031268209218979, -0.03433519974350929, 0.015412818640470505, -0.0217202790081501, 0.014701157808303833, -0.05307235196232796, 0.11292985081672668, 0.11714132130146027, -0.040593914687633514, -0.006721782963722944, 0.2812327742576599, 0.00556167820468545, 0.002247093478217721, -0.025229839608073235, 0.11261788755655289, 0.1094202920794487, -0.022480683401226997, 0.0022312516812235117, 0.03486163169145584, 0.013287585228681564, 0.00656580226495862, 0.01938057132065296, -0.017791520804166794, 0.009392947889864445, -0.06839743256568909, -0.11292985081672668, -0.035972993820905685, -0.01327783614397049, 0.07424669712781906, -0.04000898823142052, -0.008154853247106075, -0.18608468770980835, -0.0032780268229544163, -0.0868811160326004, 0.008096360601484776, 0.1081724464893341, 0.004223658237606287, -0.12166475504636765, 0.03511510044336319, 0.05743980407714844, -0.08188974112272263, -0.04991374537348747, -0.0866471454501152, -0.019146600738167763, -0.08508733659982681, -0.007964751683175564, 0.008457065559923649, 0.010167975910007954, 0.0048524546436965466, -0.038644157350063324, 0.09202846884727478, -0.05233144387602806, 0.010655415244400501, -0.09382224828004837, 0.01086988765746355, 0.003938506357371807, -0.002282432746142149, 0.2416137307882309, -0.019507305696606636, -0.012897633947432041, -0.027121102437376976, -0.014798645861446857, 0.1291518211364746, -0.11238391697406769, -0.0034656908828765154, -0.0029051359742879868, 0.011152602732181549, 0.05307235196232796, -0.1511450558900833, 0.0439474955201149, -0.0017815892351791263, 0.013560551218688488, -0.03493962064385414, -0.10770450532436371, -0.09124856442213058, -0.038215212523937225, 0.0656287744641304, 0.0009505058987997472, -0.0028832012321799994, 0.003543680999428034, -0.011874012649059296, -0.007121482864022255, -0.001653636572882533, -0.006395198870450258, 0.006283087655901909, -0.20339851081371307, 0.0008712970884516835, -0.005746904760599136, 0.015822267159819603, -0.02010198123753071, 0.010557927191257477, 0.004179788753390312, 0.27436962723731995, 0.014720655977725983, 0.14420393109321594, -0.00717022642493248, -0.03051367774605751, 0.00400918535888195, 0.10754852741956711, 0.26610267162323, -0.004287025425583124, -0.14404794573783875, -0.0010973468888550997, -0.0003805070009548217, -0.005079113412648439, 0.0077697765082120895, 0.002900261664763093, 0.005322833079844713, -0.03796174377202988, 0.01333632878959179, 0.028037486597895622, -0.003499811515212059, -0.02388450689613819, 0.06870938837528229, -0.014730404131114483, -0.1974712610244751, 0.011347577907145023, 0.0060296193696558475, 0.013950502499938011, -0.008208471350371838, 0.0019278209656476974, -0.0006574332364834845, -0.02831045351922512, -0.07572851330041885, 0.060676395893096924, -0.017918255180120468, -0.0022641539108008146, 0.01422346755862236, -0.014408694580197334, -0.009558677673339844, -0.029070857912302017, -0.05318933725357056, -0.031956497579813004, -0.009612295776605606, -0.0028417690191417933, 0.0025029988028109074, 0.013541053049266338, -0.07841917127370834, -0.0004280323046259582, -0.08391748368740082, 0.00179255660623312, -0.011825268156826496, 0.0017815892351791263, 0.005439818371087313, 0.02858341857790947, 0.016894632950425148, -0.13804270327091217, -0.0028637037612497807, 0.0006805866141803563, 0.23412667214870453, 0.11207196116447449, -0.0027832763735204935, -0.013433816842734814, 0.015968499705195427, 0.006487811915576458, -0.012429692782461643, -0.11277387291193008, -0.01536407507956028, 0.010450690984725952, -0.0436355322599411, 0.013316831551492214, -0.006326957140117884, -0.002563928719609976, 0.131959468126297, 0.10123131424188614, -0.13734079897403717, 0.0655507892370224, 0.10723656415939331, -0.08625718951225281, -0.07693736255168915, -0.008364452049136162] With ANDROID [0.017560526728630066, -0.0025206489954143763, -0.001565065118484199, -0.004763653036206961, -0.006009773351252079, 0.04073874652385712, 0.016540179029107094, 0.20969343185424805, 0.013223133981227875, 0.20780882239341736, 0.005943980999290943, -0.0026168033946305513, -0.006616274360567331, -0.01967291533946991, -0.004560244735330343, 0.0022333874367177486, -0.0006560852052643895, 0.006724789272993803, 0.007219155319035053, -0.006807046011090279, 0.059210970997810364, 0.008006574586033821, -0.0298028364777565, 0.0026111663319170475, -0.06040123477578163, -0.003707980504259467, 0.0029578227549791336, -0.03018888272345066, 0.1781667321920395, 0.011050385423004627, 0.005854886025190353, 0.017274057492613792, -0.09191303700208664, -0.006212593521922827, -0.2262837141752243, -0.15064898133277893, 0.1591326892375946, -0.006933901458978653, -0.001802279381081462, -0.10352537781000137, -0.0010443583596497774, 0.0018612570129334927, 0.006398417986929417, 0.0012554702116176486, 0.0074211121536791325, 0.010717593133449554, 0.049051254987716675, 0.24959337711334229, 0.006417354568839073, -0.057666826993227005, -0.0468457005918026, -0.002862254623323679, -0.00486677186563611, -0.002375381998717785, -0.01682371459901333, 0.009011342190206051, 0.12074223905801773, 0.001989581622183323, -0.009173679165542126, -0.013144731521606445, -0.015070175752043724, -0.04363885894417763, -0.0003053119871765375, -0.006338771432638168, 0.004932073410600424, -0.15332861244678497, -0.004816148895770311, -0.0023492116015404463, 0.016545282676815987, -0.001709619304165244, -0.013169904239475727, -0.11412511020898819, -0.202517569065094, 0.009383699856698513, 0.1297224462032318, 0.0056001124903559685, -0.01269744522869587, 0.0021330846939235926, -0.08546653389930725, -0.1334666609764099, -0.0027801108080893755, -0.07388244569301605, 0.00505793234333396, 0.07371390610933304, 0.03501145541667938, -0.0006730420864187181, -0.002135098446160555, 0.004249595105648041, -0.036232441663742065, 0.263942152261734, 0.0027228561230003834, -0.0020696944557130337, 0.0017612290102988482, -0.002822405658662319, 0.1190698966383934, 0.04764315485954285, 0.02588491700589657, -0.09784920513629913, -0.0023657327983528376, -0.02211504615843296, 0.004360306076705456, -0.0026493342593312263, -0.001022880314849317, 0.00408570934087038, -0.0034248982556164265, 0.003120097564533353, -0.07066097110509872, 0.003962109796702862, -0.020672880113124847, 0.002750005107372999, -0.10889820009469986, -0.0069341775961220264, 0.010038419626653194, 0.1742521971464157, 0.0003855641698464751, -0.0016570232110098004, 0.012623189948499203, 0.007401735056191683, 0.030959144234657288, -0.04500763118267059, -0.018015660345554352, -0.004637634847313166, 0.1341673582792282, -0.002675049938261509, 0.0010797850554808974, -0.004457265138626099, 0.0026313276030123234, 0.0004421667836140841, 0.02306954190135002, 0.21203938126564026, -0.0042820824310183525, 0.0033936691470444202, -0.004675364587455988, -0.006049075163900852, -0.08900583535432816, -0.002515271073207259, -0.06748009473085403, 0.02633572369813919, -0.007937341928482056, -0.0005898139788769186, -0.000987825682386756, -0.0029705087654292583, -0.0008315134327858686, -0.26069924235343933, 0.0367228239774704, 0.08291738480329514, -0.012987925671041012, -0.00524356821551919, 0.010866782627999783, -0.0017442327225580812, 0.0024614043068140745, -0.24548490345478058, -0.0642000138759613, 0.009642115794122219, 0.0014905520947650075, 0.004344462417066097, -0.001906131743453443, 0.008299516513943672, 0.05748245120048523, 0.0031748428009450436, 0.0020936853252351284, 0.0012538537848740816, 0.004880981519818306, -0.0021094498224556446, 0.0012627202086150646, -0.0025581330992281437, -0.0043145096860826015, 0.17136064171791077, -0.0027595553547143936, 0.0014381170039996505, 0.1392369121313095, 0.13462324440479279, -0.0060705929063260555, 0.023846164345741272, -0.041592974215745926, 0.0018410480115562677, 0.12148106843233109, -0.03168937936425209, -0.00026864526444114745, -0.005666567012667656, 0.0920085608959198, 0.026908613741397858, 0.0023466504644602537, 0.0051388428546488285, -0.11932720988988876, -0.19283035397529602, 0.024459129199385643, -0.06673143804073334, 0.08777308464050293, 0.01421852596104145, -0.005940374452620745, 0.007013058289885521]

this is my code

imglib.Image _convertCameraImage(CameraImage image, CameraLensDirection _dir) {
    print('_convertCameraImage');
    if( GetPlatform.isAndroid == true ){
      print(' Android');
      int width = image.width; print('11'); print(width);
      int height = image.height; print('12'); print(height);
      var img = imglib.Image(width, height); print('13');
      const int hexFF = 0xFF000000; print('14');
      print( 'image.planes.length=>' + image.planes.length.toString());
      print(image.planes[0].bytesPerRow);
      final int uvyButtonStride = image.planes[1].bytesPerRow; print('15');
      final int uvPixelStride = image.planes[1].bytesPerPixel; print('16');
      for (int x = 0; x < width; x++) {
        for (int y = 0; y < height; y++) {
          final int uvIndex = uvPixelStride * (x / 2).floor() + uvyButtonStride * (y / 2).floor();
          final int index = y * width + x;
          final yp = image.planes[0].bytes[index];
          final up = image.planes[1].bytes[uvIndex];
          final vp = image.planes[2].bytes[uvIndex];
          int r = (yp + vp * 1436 / 1024 - 179).round().clamp(0, 255);
          int g = (yp - up * 46549 / 131072 + 44 - vp * 93604 / 131072 + 91).round().clamp(0, 255);
          int b = (yp + up * 1814 / 1024 - 227).round().clamp(0, 255);
          img.data[index] = hexFF | (b << 16) | (g << 8) | r;
        }
      }
      print('17');
      var img1 = imglib.copyRotate(img, -90);
      print(img1.toString());
      return img1;
    }else{
      print(' IOS');
      imglib.Image img = imglib.Image.fromBytes(
        image.width,
        image.height,
        image.planes[0].bytes,
        format: imglib.Format.bgra,
      );
      var img1 = (_dir == CameraLensDirection.front)
          ? imglib.copyRotate(img, -90)
          : imglib.copyRotate(img, 90);
      print('retorno');
      print(img1.toString());
      return img1;
    }

  }

I appreciate your help, thanks.

[Rajatkalsotra]

@fnoceda @doanbh Since the input to the model will be different in android and IOS due to format of the image , so the output embeddings would be different in both the cases. Workaround will be to save the embeddings of both OS and use the one as required. Thanks and create a pull request to repo if you like.