search

puffinsoft / jscanify

Open-source Javascript mobile document scanner.
https://puffinsoft.github.io/jscanify/
MIT License
940 stars 51 forks source link

WebRTC is very complicated for photos #13

Closed adantart closed 8 months ago

adantart commented 1 year ago

As you know, capturing a photo is basically taking a still frame from the video stream.

When doing video, usually any project is compatible with the idea of wanting a 360, 720 or 1080 video... but when you're working with photos, you usually want the best possible photo resolution.

There is a "WebRTC resolution scanner" but it works pretty poorly. For example, on my iPhone 14, it gives me "fail" as a result in all resolutions, making it difficult for me to set up a stream with "the best resolution the mobile can offer".

And this may not be a problem when you use WebRTC to take a profile picture or a picture where good resolution is not required, but in the case of a document scanner, it is critical to have the best possible resolution to be able to OCR it.

How do you deal with this problem and how do you configure WebRTC to take the best possible mobile resolution?

I have tried with the "ideal" constraints but without good results.

Thanks in advance

ColonelParrot commented 1 year ago

Interesting. From what I gather, you are having trouble getting high resolution images through WebRTC? Could you share the code you're using? Might be able to help from there.

adantart commented 1 year ago

It's just to have the getUserMedia with a constraints of 1920 or higher... OpenCV Mat has a known bug with garbage collector (https://github.com/opencv/opencv/issues/15060) and when you use video or high resolution photo is too heavy for Javascript. I'm trying to use to capture more than 1 document, and if the "session" lasts more than 3/4 minutes, it finally crashes resulting in a memory fault. I've tried to "delete" everything possible, but it's something very "deep" in OpenCV.

ColonelParrot commented 1 year ago

OK, so you are able to get high res images, but are experiencing memory leaks?

ColonelParrot commented 1 year ago

Hmmmm, ok. If it's a problem on OpenCV's side then there's not much I can do, but I'll take a look to see whether there could be some fault on my side exacerbating this problem. I wonder whether you could simply set a timer on the camera? E.g., after 45 seconds automatically close the stream due to inactivity.

adantart commented 1 year ago

I already tested it. The problem is that OpenCV does not "clean" well. The problem is not from the streaming that "overloads" the memory, but from each time you call OpenCV to do the calculations. That is, it is as if you have a "maximum number of seconds where OpenCV works". Even if you cut the streaming, it doesn't minimize the overall problem. I'll keep thinking of something.

Of course, all this happens when you want to make a "long" and continuous use of OpenCV with resolutions beyond 800x600.