search

capacitor-community / barcode-scanner

A fast and efficient (QR) barcode scanner for Capacitor
MIT License
437 stars 172 forks source link

Idea: Try an implementation with Googles ML Kit #107

Closed thegnuu closed 1 year ago

thegnuu commented 2 years ago

Due to some issues which are hard to resolve in a cross-platform manner, it might make sense to create a PoC with ML Kit.

https://developers.google.com/ml-kit/vision/barcode-scanning

It might be absolutely possible that after the PoC we find out that it is not working better, but at it least it is worth trying since it should not be too much work to create a test plugin :)

JanMisker commented 2 years ago

@thegnuu Looks interesting! The samples provided in the MLKit docs are already pretty close to what the plugin should provide, so indeed should be pretty doable to implement a PoC.

thegnuu commented 2 years ago

Indeed yes, I already played around with the example app and the results are really promising so far.

It will take a few hours to create a PoC though, at least for me since I am not as fluent in Swift/Java as in other languages, but I should find some time in the next two weeks to tinker around with it 👍🏼

JanMisker commented 2 years ago

It will take a few hours to create a PoC though, at least for me since I am not as fluent in Swift/Java as in other languages, but I should find some time in the next two weeks to tinker around with it 👍🏼

I'm quite comfortable with Swift; if you can do an Android PoC implementation I could use that as a guide to make the iOS one.

thegnuu commented 2 years ago

Sounds great! I spent some time today to read some examples how all of this has to be done...

Unfortunately I was not able to start working on it, since I was not really sure where to start... I think the easiest way will be to create a completely new plugin to work on this one and move it to this repo once most of the work is done...

This seems to be a good and simple example to start the android implementation.

phal0r commented 2 years ago

Pretty cool idea. Wouldn't this also allow to detect multiple codes at the same time and send and update the position of the detected codes? In combination with the overlayed webview, this would allow to augment the barcodes with related content within the webview.

thegnuu commented 2 years ago

@phal0r Exactly yes! This is one of the reasons why I think this is a good idea. It would allow us to detect any barcode within the camera frame and report back all of those with the position where they are detected.

phal0r commented 2 years ago

@thegnuu yeah, so what does it need to move this forward?

thegnuu commented 2 years ago

@phal0r Right now, everything ;)

I played around with the demo projects of ML Kit and thought about implementing a simple PoC first, but I realised that once the PoC is implemented most of the critical parts are already done...

As I already told, I am not often developing native code. @JanMisker offered help on the iOS/Swift part of the implementation once the Android part is done, so I will try to focus on the Android part. If you @phal0r have some Android/Java/Kotlin experience and could offer some help, that would totally speed up the process. I have a lot of other projects running at the moment and it might take me some weeks to get this done (especially because I am not really fast at coding this stuff at the moment)...

I can create a branch this week and publish the (in my opinion) required interfaces changes for the new implementation. This is the only thing I was currently able to prepare.

phal0r commented 2 years ago

@thegnuu yeah, let me know, when the interfaces are available. This will certainly help :)

thegnuu commented 2 years ago

Of course, I will do this this week and let you know!

I think we can reduce the current functions and remove the "single scan" ability, since this can be easily implemented by each developer itself.... also things like hideBackground and showBackground do not really make sense, since they are called by the plugin anyways.

thegnuu commented 2 years ago

@phal0r & @JanMisker I just pushed the branch ml-kit for this changes...

The only thing I updated right now is the typescript definitions part: https://github.com/capacitor-community/barcode-scanner/blob/ml-kit/src/definitions.ts

What do you think about this changes?

phal0r commented 2 years ago

@thegnuu yeah, makes sense to me.

So in addition my general considerations:

Regarding these considerations, the interface makes a lot of sense, so I guess, you would agree to my thoughts, correct?

thegnuu commented 2 years ago

@phal0r exactly my thoughts yes. In my opinion we should keep the native side quite simple and let the individual implementation handle parts like deduplication and when the scanning is paused, stopped aso.

ML Kit will generate a lot of events, so a small throttling on native side might be required, but I think we will see this once we have a simple test environment...

danielberebel commented 2 years ago

Hi, I found other package on development (apparently abandoned) for capacitor ml kit barcode scanner. https://github.com/franckysolo/capacitor-mlkit-barcodescanner

And other for cordova. https://github.com/Ratsoc/cordova-plugin-firebase-ml-kit-barcode-scanner

I hope it can help you . And although I have no experience with native development I want to offer my help for anything.

thegnuu commented 2 years ago

@danielberebel sorry for my delayed response and thank you for pointing to that examples, this will surely help to create at least the Android part of the plugin.

If you would like to help you can try to start the rewrite of the Android plugin - I won't find time until next week to get started, if there is already some work done I can pick up the work from there :)

phal0r commented 2 years ago

@thegnuu Is there a branch somewhere already?

thegnuu commented 2 years ago

@phal0r Kind of: https://github.com/capacitor-community/barcode-scanner/tree/ml-kit

Currently only the new interfaces are committed, I am currently working on it and trying to get at least a single code scanned on Android.... But it's not going forward really fast since I have to look up a lot of stuff ;)

Taufeeq-Mowzer commented 2 years ago

Hi, I'm not experienced at all but I see that this has been done with flutter and was wondering if you could use their Java/Ios code to help form your bindings. Github repo of the most used barcode scanner implementation Github repo of the flutter barcode scanner using firebase ML Kit

thegnuu commented 2 years ago

I was finally able to get ML Kit working (I am currently only working on the iOS version) and so far the results are really promising. I need to do some cleanup but in a week or so I should be ready to provide a iOS test build.

Would anyone out there be willing to support the Android development once the iOS version is done as an example?

smachi commented 2 years ago

It would be awesome to have an MLKit implementation for this plugin as the current version is not working well for us (it isn't able to scan some barcodes...)

ferranJS commented 2 years ago

Would anyone out there be willing to support the Android development once the iOS version is done as an example?

Hi I managed to make a quick little iOS app (permissions are not given yet so had to make a workaround) and the results are very good so I'm looking forward to starting with coding the Android version. If anyone wants to join me in the implementation, I'm open to get some help! as I'm not very experienced in Android development.

1andonlyziz commented 2 years ago

@fewNeurons I'm currently working on android implementation the results are quite promising , I need to do some code clean up and then I'll Create a PR for it.

My next milestone is drawing rectangles around the barcodes that was scanned and detected on the camera View

thegnuu commented 1 year ago

@1andonlyziz sounds great! Were you able to make any progress or do you need any help somewhere? :)

1andonlyziz commented 1 year ago

@thegnuu yeah, I created a PR on ml-kit branch, please review and take a look, I can make a demo if you want ^_*

the only thing that is missing is drawing graphic objects on the barcode view when something is scanned with startScanning function

thegnuu commented 1 year ago

@1andonlyziz awesome! Thank you very much for your work! I will have a look at it later today!

I don't think that this feature is necessary to begin with to be honest, surely we can think about it later. :)

thegnuu commented 1 year ago

Hi everyone!

Thanks to the great work of @1andonlyziz the ml-kit branch should now work on android as well! So far, the results are really promising!

There are still a few TODOs and some cleanups that need to be done, but I should be able to create a dev-release in the upcoming week or so!

PetervLeeuwen commented 1 year ago

Hey @thegnuu , when do you think the ml-kit branch will be merged to master and available in the official npm repo? I would love to start using this feature! 😄

thegnuu commented 1 year ago

@PetervLeeuwen I would love to release it as soon as possible as well, unfortunately, I am way too busy at the moment to provide a proper ETA, I really hope I can finish the open work by the end of this month, but I cannot promise anything. I will do my best though.

JanMisker commented 1 year ago

Hi @thegnuu I'm more than happy to help out. I am testing the branch and it seems to work quite well already. I do see a couple of small things, should I just make issues for each separate? And submit PRs if I have them? Or should we coordinate more? Not sure how much double work there would be... For now I see:

thegnuu commented 1 year ago

v5.0.0-beta.1 is available now :) This is an initial pre-release of the ml-kit implementation and should help anyone who is willing to test the new implementation with an easier setup ;)

thegnuu commented 1 year ago

Closing this for now, maybe a discussion would make more sense for the future :)