search

juliansteenbakker / mobile_scanner

A universal scanner for Flutter based on MLKit. Uses CameraX on Android and AVFoundation on iOS.
BSD 3-Clause "New" or "Revised" License
756 stars 446 forks source link

Add scanWindow to optionally limit scan area #176

Closed casvanluijtelaar closed 1 year ago

casvanluijtelaar commented 1 year ago

Implements https://github.com/juliansteenbakker/mobile_scanner/issues/11 for IOS, Android & MacOS

juliansteenbakker commented 1 year ago

Thank you for your contribution! Can you please run flutter format . and push the changes?

casvanluijtelaar commented 1 year ago

just a heads up, I can confirm the new functionality on Android. But I can't seem to be able to can anything on IOS, using a scanWindow or not, and I'm not sure if that's because of something added in this pull. since Scanwindow code is completely skipped over if not provided.

casvanluijtelaar commented 1 year ago

this is the furthest I'm going to get with this without external help, 2 things missing from IOS:

@juliansteenbakker or anyone else ;)

juliansteenbakker commented 1 year ago

Thank you for all your work! I will look into this to see if i can get AVCaptureMetadataOutput flow integrated.

ayavn commented 1 year ago

@casvanluijtelaar I had an app crash when using scanWindow.

java.lang.IllegalArgumentException: Image dimension, ByteBuffer size and format don't match. Please check if the ByteBuffer is in the decalred format.

Device: Samsung Galaxy A22. Android 12.

flutter doctor 

[✓] Flutter (Channel stable, 3.0.1, on macOS 11.6 20G165 darwin-arm, locale en)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.1)
[✓] VS Code (version 1.67.2)
[✓] Connected device (4 available)
[✓] HTTP Host Availability

• No issues found!
casvanluijtelaar commented 1 year ago

big change in approach, instead of actively cropping just compare the barcode locations to the frame. the only thing that needs to happen now is make sure the Rect dimensions coming from flutter match the android and ios rects

casvanluijtelaar commented 1 year ago

@juliansteenbakker I'd like some input on how to approach the Rect implementation, specifically because the screen size in flutter and the video size don't reflect each other 

flutter screen size: Size(428.0, 926.0)
video size: Size(3024.0, 4032.0)

from a flutter perspective I'd intuitively want to use the flutter logical pixel size, so that a Rect for the camera cutout is a 1 to 1 match for a rect i'd perhaps want to overlay in a custom paint.

but that of course makes for a trickier conversion. I guess we could wait on the flutter side till we have the video width and height and then converting that to a screen size.

similar how it's done in the flutter widget itself :

image
casvanluijtelaar commented 1 year ago

shouldn't the MobileScannerArguments size be in flutter logical pixel size too? that would make life easier. knowing the actual camera capture size seems pretty useless to me

casvanluijtelaar commented 1 year ago

and how would that work with a custom BoxFit

casvanluijtelaar commented 1 year ago

something like this:

  /// for boxfit.contain
  Size get actualSizeOfTextureInFlutterCoordinates {
    /// currently full screen size but should probably be the actual MobileScanner widget size
    final screenWidth = window.physicalSize.shortestSide / window.devicePixelRatio;
    final screenHeight = window.physicalSize.longestSide / window.devicePixelRatio;

    final screenSize = Size(screenWidth, screenHeight);

    final ratio = screenSize.longestSide / screenSize.longestSide;

    return Size(size.width * ratio, size.height * ratio);
  }
casvanluijtelaar commented 1 year ago

the coordinate grids are extremely inconsistent.

and then the flutter logical pixel size of the widget is different yet. how do we convert the incoming Rect so that its consistent, and aligned, on all platforms?

@juliansteenbakker how did you do this for qr_code_scanner?

casvanluijtelaar commented 1 year ago

I feel like I'm doing a terrible job explaining the issue:

image

any Rect defined in flutter will have it's coordinates relative to the widget size. But for the native barcode scanning, we need a rect relative to the image it scans.

This is not an easy thing to do. first of all because depending on the boxfit the scan area could fall partly outside the texture, the texture couldn't have the same aspect ratio, ... etc.

And on top of that, as mentioned earlier, texture size != actual image size used by Mlkit. so the coordinates would have to be scaled to match.

casvanluijtelaar commented 1 year ago

Progress, this might be a functional implementation on the flutter side to overcome all these issues:

  void calculateScanWindowRelativeToTexture(BoxFit fit, Rect scanWindow, Size textureSize, Size widgetSize) {

    /// map the texture size to get its new size after fitted to screen 
    final fittedTextureSize = applyBoxFit(fit, textureSize, widgetSize).destination;
    final minX = widgetSize.width / 2 - fittedTextureSize.width / 2;
    final minY = widgetSize.height / 2 - fittedTextureSize.height / 2;
    final textureWindow = Rect.fromLTWH(minX, minY, fittedTextureSize.width, fittedTextureSize.height);

    /// create a new scan window and with only the area of the rect intersecting the texture
    final scanWindowInTexture = scanWindow.intersect(textureWindow);

    /// update the scanWindow left and top to be relative to the texture not the widget
    final newLeft = scanWindowInTexture.left - minX;
    final newTop = scanWindowInTexture.top - minY;

    /// new scanWindow that is adapted to the boxfit and relative to the texture
    final scanWindowRelativeToTexture = Rect.fromLTWH(newLeft, newTop, scanWindowInTexture.left, scanWindowInTexture.top);

    /// get the scanWindow as a percentage of the texture
    final percentageLeft =  scanWindowRelativeToTexture.left / fittedTextureSize.width;
    final percentageRight = scanWindowRelativeToTexture.right / fittedTextureSize.width;
    final percentageTop = scanWindowRelativeToTexture.top / fittedTextureSize.height;
    final percentagebottom = scanWindowRelativeToTexture.bottom / fittedTextureSize.height;

    /// this rectangle can be send to native code and used to cut out a rectangle of the scan image
    final percentageRelativeRect = Rect.fromLTRB(percentageLeft, percentageRight, percentageTop, percentagebottom);
  }

this returns a Rect that contains percentage data of its position on the texture. now in native code we can use these percentages to reconstruct a scan area in native code. (assuming the texture aspect ratio is the same as the scan image aspect ratio)

image
casvanluijtelaar commented 1 year ago

Almost there, just a couple bugs in the previously mentioned method left

casvanluijtelaar commented 1 year ago

I can confirm It's working correctly now! maybe we should rename the properties to something like scanArea or something else instead of the current scanWindow

juliansteenbakker commented 1 year ago

Thank you for all your work and explanation! I've tested it on both android and iOS, and while i had no problems on iOS, i did have problems on Android. I tried it with 2 devices however it seems like the scanning area expands upwards of the RECT. Also, it seems quite hard to scan something in general on Android rather that on iOS. Do you have the same problems or is everything working for you on Android?

casvanluijtelaar commented 1 year ago

Thank you for all your work and explanation! I've tested it on both android and iOS, and while i had no problems on iOS, i did have problems on Android. I tried it with 2 devices however it seems like the scanning area expands upwards of the RECT. Also, it seems quite hard to scan something in general on Android rather that on iOS. Do you have the same problems or is everything working for you on Android?

I can't replicate that with the example I added in the pull request, are you using a different setup?

casvanluijtelaar commented 1 year ago

about the slower scanning on Android I notice the same, but the Rect seems to work.

but the image that is scanned for android is (for me): 640 x 480 whilst on Iphone it is (for me): 1284 x 1712

if you for example have a full screen camera on android and you cut-out a centered rect, you're going to have an area of less than 100x100 pixels to find and recognize a scan code, Is there any reason the android image is this low quality?

the preview resolution on android is 1440x1080, that would be fine

casvanluijtelaar commented 1 year ago

Idk anymore, android is killing me. on some devices the imageAnalysis will have a different size and rotation than the preview. and I can't find any way to dynamically set that properly. and even then from the documentation "the analyzer might not use the provided size".

something like this:

   val analysisBuilder = ImageAnalysis.Builder()
              .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
              .setTargetResolution(preview!!.resolutionInfo?.resolution ?: Size(640, 480))

Sidenote your ratio implementation doesn't work..., you can't send a dart Ratio object through a method channel without serialization

@juliansteenbakker if you want to look into this awesome, I'm wasting to many hours on this at the moment. preview!!.resolutionInfo?.resolution works on some devices (onplus 7 pro) but will return null on other (pixel 5)

casvanluijtelaar commented 1 year ago

issue:

  val previewBuilder = Preview.Builder()
  if (ratio != null) { previewBuilder.setTargetAspectRatio(ratio) }

  preview = previewBuilder.build().apply { setSurfaceProvider(surfaceProvider) }

  val analysisBuilder = ImageAnalysis.Builder()
          .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
          /// preview!!.resolutionInfo?.resolution is still null at this point so we can't use it.
          .setTargetResolution(preview!!.resolutionInfo?.resolution ?: Size(1920, 1080))

  if (ratio != null) { analysisBuilder.setTargetAspectRatio(ratio) }
  val analysis = analysisBuilder.build().apply { setAnalyzer(executor, analyzer) }

  val selector = if (facing == 0) CameraSelector.DEFAULT_FRONT_CAMERA else CameraSelector.DEFAULT_BACK_CAMERA

  /// preview resolution is defined at the same time as the analyzer is set, so there's no way to have them
  /// match resolutions up from. at this point the above code will fall back to Size(1920, 1080) which can in no way 
  /// guarantee the same resolution and aspect ratio and therefore matching Rects. IOS matches resolutions
  /// automatically so this isn't an issue
  camera = cameraProvider!!.bindToLifecycle(activity as LifecycleOwner, selector, preview, analysis)

  val analysisSize = analysis.resolutionInfo?.resolution ?: Size(0, 0)
  val previewSize = preview!!.resolutionInfo?.resolution ?: Size(0, 0)

  Log.i("LOG", "Analyzer: $analysisSize")
  Log.i("LOG", "Preview: $previewSize")
juliansteenbakker commented 1 year ago

Sorry again for the late response and thanks for all the hard work! I'll try and see if i can get some other devices to test it on and see if this maybe is an issue with the camera2 library

casvanluijtelaar commented 1 year ago

I just started working on this again today @juliansteenbakker ;). was just about to push when I noticed you merged and started working on it. found the root of the issue, ... well multiple.

have a look at my branch for latest updates, the latest update functions fully on my devices, but I have manually flipped the image for scanning, and that should be fixed ;)

juliansteenbakker commented 1 year ago

Nice! So i merged your changes in a local branch so i can publish it as a beta. If you want you can create a new pull request and merge it against feature/scan-window instead of master. After that i will publish it as a beta release!

mehdiwaysi commented 1 year ago

This feature is very critical, please add it

vincnet500 commented 1 year ago

Could it be possible to release the beta version ?

weili1210 commented 1 year ago

Hi, just want to ask is this feature available on beta version or not?

ArslanAsghar123 commented 1 year ago

Hi all, is there any possible way to add texts inside the overlay?