usdlCombinedRecognizer and mrtdRecognizer

[AlibekJ]

So, when two recognizers are used at once like

var recognizerCollection = new cordova.plugins.BlinkID.RecognizerCollection([usdlCombinedRecognizer, mrtdSuccessFrameGrabber]);

plugin works fine with US DL: reads both sides, etc. But when presented a passport, it reads the first side and then demands for a back side, which, obviously, non existent in passports.

I guess desired behavior is to get plugin stop scanning once it sees a recognizable passport.

[culoi]

Hello Alibek,

Can you please share which exact Recogniozers are you using?

I see that you are setting mrtdSuccessFrameGrabber. SuccessFrameGrabber is a special Recognizer that wraps some other Recognizer and impersonates it while processing the image.

Did you mean to use MrtdRecognizer to read passport? MrtdRecognizer is used for scanning and data extraction from the Machine Readable Zone (MRZ) of the various Machine Readable Travel Documents (MRTDs) like ID cards and passports.

Best regards

[AlibekJ]

Hi Ivan,

Here is what I am trying to achieve: Once I start the plugin and it sees US DL, it should scan the front side, ask for the back side, return data and stop. If it sees a passport, it should just return data and stop.

Currently, the behavior is the same regardless of the document it sees: Gets front side, asks for the back side and does not return until it got the back side. US DL has two sides, while passport has only one. You see my problem?

Each of the recognizers work just fine when used alone.

Workaround is to ask client what document he is about to show and only then start the plugin appropriately. But I would like to avoid this.

Here is my code:

    var mrtdRecognizer = new cordova.plugins.BlinkID.MrtdRecognizer();
    mrtdRecognizer.returnFaceImage = true;
    mrtdRecognizer.returnFullDocumentImage = true;
    //var mrtdSuccessFrameGrabber = new cordova.plugins.BlinkID.SuccessFrameGrabberRecognizer(mrtdRecognizer);

    var usdlCombinedRecognizer = new cordova.plugins.BlinkID.UsdlCombinedRecognizer();
    usdlCombinedRecognizer.returnFaceImage = true;
    usdlCombinedRecognizer.returnFullDocumentImage = true;

    var documentOverlaySettings = new cordova.plugins.BlinkID.DocumentVerificationOverlaySettings();

    var recognizerCollection = new cordova.plugins.BlinkID.RecognizerCollection([mrtdRecognizer, usdlCombinedRecognizer]);
    //tried this as well
    //var recognizerCollection = new cordova.plugins.BlinkID.RecognizerCollection([mrtdSuccessFrameGrabber, usdlCombinedRecognizer]); 

Can you suggest a way to achieve the desired behavior?

[jcular]

Hi @AlibekJ,

does this happen on Android or iOS, or both? What kind of document are you trying to scan using MRTD recognizer?

You can set allowMultipleResults to true in your RecognizerCollection, this should make it possible to scan MRTD and USDL's with your setup.

This probably happens because the MRTD you are trying to scan contains a face image, which makes the front part of the USDLCombined recognizer valid. Since it becomes valid before MRTD recognizer, it switches to scanning the back side of the USDL and MRTD recognizer won't become valid. By setting allowMultipleResults to true, you can allow MRTD recognizer to become valid even though USDL front becomes valid too.

Kind regards, Jure

[AlibekJ]

Hi Jure,

Thanks for the suggestion to try allowMultipleResults, but I am still having issues. Here is the code which ended up working SOMETIMES:

Sometimes it scans a passport and happily exists, but sometimes, about 1 in 10 times, it demands a back side. Please, help.

    var mrtdRecognizer = new cordova.plugins.BlinkID.MrtdRecognizer();
    mrtdRecognizer.returnFaceImage = true;
    mrtdRecognizer.returnFullDocumentImage = true;
    mrtdRecognizer.allowMultipleResults = true;

    var mrtdSuccessFrameGrabber = new cordova.plugins.BlinkID.SuccessFrameGrabberRecognizer(mrtdRecognizer);
    mrtdSuccessFrameGrabber.allowMultipleResults = true;

    var usdlCombinedRecognizer = new cordova.plugins.BlinkID.UsdlCombinedRecognizer();
    usdlCombinedRecognizer.returnFaceImage = true;
    usdlCombinedRecognizer.returnFullDocumentImage = true;
    usdlCombinedRecognizer.allowMultipleResults = true;

    var documentOverlaySettings = new cordova.plugins.BlinkID.DocumentVerificationOverlaySettings(); //  DocumentOverlaySettings DocumentVerificationOverlaySettings();
    documentOverlaySettings.allowMultipleResults = true;
    var usdlSuccessFrameGrabber = new cordova.plugins.BlinkID.SuccessFrameGrabberRecognizer(usdlCombinedRecognizer);
    var recognizerCollection = new cordova.plugins.BlinkID.RecognizerCollection([mrtdSuccessFrameGrabber, usdlSuccessFrameGrabber]); //mrtdSuccessFrameGrabber   usdlCombinedRecognizer

    recognizerCollection.allowMultipleResults = true;

    scanButton.addEventListener('click', function () {
      cordova.plugins.BlinkID.scanWithCamera(
        function callback(cancelled) {},
        function errorHandler(err) {},
        documentOverlaySettings, recognizerCollection, licenseKeys
      );
    });

Tried on both Android and iOS. Scanning generic ICAO MRZ passports and US driving licenses.

Here are some examples of a passport it often reads as DL one two three four five

[jcular]

Hi @AlibekJ,

this will happen if usdlCombinedRecognizer gets triggered before mrtdrecognizer, because for the front side it will detect a face. Increasing the numStableDetectionThreshold(default value is 6) might help, it might give more time to MRTDRecognizer to read the mrz if it's a bit hard to read.

Kind regards, Jure

[AlibekJ]

Hi Jure,

thanks, will try.

Can't you guys put an image classifier to work and let it decide which recognizer to use? I believe openCV now has classifier built-in these days and it might even be hardware accelerated.

Alternatively, is there a way to pass images to a callback, so I could put my own classifier?

For exapmle: you pass down the success image to the callback function and if it returns true you carry on doing what you were doing and if it returns false, discard the image and use the other recognizer.

Alibek

[jcular]

Hi @AlibekJ,

we are currently working on that. We don't support passing images to a callback in phongeap, although we do provide a way to setup your own camera sessions, but only on native platforms(ios, android).

In theory you could do that on ios or android, but you'd have to train your own classifier, which requires a large dataset.

Although increasing the numStableDetectionsThershold might help, it's still might not be as reliable as having two entry points, one for scanning passports and the other for scanning usdls.

Kind regards, Jure

[AlibekJ]

Thanks, Jure.

Wanted to avoid to, but I guess will end up doing two separate screen flows for usdl and passports. At least until you will implement the classifier.

Once classifier is trained, please, make it available for retraining. For obvious reasons, can't send you the documents we capture, but if I'll get your pre-trained classifier, I would retrain it on my data and share it back with you.