NilsEngelbach commented 3 years ago

First of all, thank you for all the efforts you put into this OS project. This should not be taken for granted!

We use the ngx-scanner in an angular application to scan various different types of codes in different sizes from different materials (printed paper, laser marked metal parts, ...). Especially the laser marked codes are tough because the codes are not super accurate, small and the metal surface might have scratches and reflect the light. So we spend quite some time improving the scanning with the ngx-scanner that maybe others would also be interested in.

What we have customized so far:

Use the highest camera resolution possible
Define / Crop the area to search for codes
Support for inverted codes (white code on black material)

We did the tweaks by overriding methods on the BrowserCodeReader and ZXingScannerComponent. Especially the first two adjustments might be something that could be considered to add as feature into the component/library directly.

Use the highest camera resolution possible

Is your feature request related to a problem? Please describe. By default it always choose a

640x480 resolution for my laptop webcam in chrome (even tough it would support 1280x720)
640x480 resolution for the IPad in safari (even tough it would support 1920x1080)

After some research we stumbled over the constraints (https://developer.mozilla.org/en-US/docs/Web/API/Media_Streams_API/Constraints) and tweaked the two places where the constraints where applied:

ZXingScannerComponent.prototype.getAnyVideoDevice = (): Promise<MediaStream> => {
  return navigator.mediaDevices.getUserMedia({
    audio: false,
    video: {
      width: { min: 640, ideal: 1920 },
      height: { min: 400, ideal: 1080 },
      aspectRatio: { ideal: 1.7777777778 }
    }
  });
};

BrowserCodeReader.prototype.decodeFromVideoDevice = async function(
  deviceId: string | undefined,
  previewElem: string | HTMLVideoElement | undefined,
  callbackFn: any,
): Promise<IScannerControls> {

  // We had to comment this out because it is a private method...
  // BrowserCodeReader.checkCallbackFnOrThrow(callbackFn);

  let videoConstraints: MediaTrackConstraints;

  if (!deviceId) {
    videoConstraints = { facingMode: 'environment' };
  } else {
    videoConstraints = {
      deviceId: { exact: deviceId },
      width: { min: 640, ideal: 1920 },
      height: { min: 400, ideal: 1080 },
      aspectRatio: { ideal: 1.7777777778 }
    };
  }

  const constraints: MediaStreamConstraints = { video: videoConstraints };

  return await this.decodeFromConstraints(constraints, previewElem, callbackFn);
};

This lead to a higher resolution and therfore better scans.

Describe the solution you'd like I think it might be useful to make the constraints customizable (as input of the component maybe), and also change the default to something like:

{
  video: {
    width: { min: 640, ideal: 1920 },
    height: { min: 400, ideal: 1080 },
    aspectRatio: { ideal: 1.7777777778 }
  }
}

Additional context

There might be a need for some further investigations for other browser, because they all tend to interpret the constraints a little bit different. Here is a nice website that lets you test common constraints for your camera: https://webrtchacks.github.io/WebRTC-Camera-Resolution/

Define / Crop the area to search for codes

Is your feature request related to a problem? Please describe. We added two different modes to our scanner to crop the scanned area for 1D and 2D codes:

This was a bit difficult because of the different resolutions / sizes and orientations of the devices but what we basically do is some calculation where the displayed cropping area will be found in the actual canvas, and fill a new canvas with only the cropped part to evaluate in the code detection:

BrowserCodeReader.createBinaryBitmapFromCanvas = function (canvas: HTMLCanvasElement): BinaryBitmap {

  const cameraCanvasImageWidth = canvas.width;
  const cameraCanvasImageHeight = canvas.height;
  const videoElementHeight = (document.querySelector('zxing-scanner video') as any).offsetHeight;
  const videoElementWidth = (document.querySelector('zxing-scanner video') as any).offsetWidth;

  let factor = 1;
  if (videoElementWidth > videoElementHeight) {
    factor = cameraCanvasImageWidth / videoElementWidth;
  } else {
    factor = cameraCanvasImageHeight / videoElementHeight;
  }

  const width = Math.floor((this.squareShape ? 250 : 450) * factor);
  const height = Math.floor((this.squareShape ? 250 : 200) * factor);

  const left = (cameraCanvasImageWidth - width) / 2;
  const top = (cameraCanvasImageHeight - height) / 2;

  const croppedCanvas = document.createElement('canvas');
  croppedCanvas.width = width;
  croppedCanvas.height = height;
  const croppedCanvasContext = croppedCanvas.getContext('2d');
  croppedCanvasContext.rect(0, 0, width, height);
  croppedCanvasContext.fillStyle = 'white';
  croppedCanvasContext.fill();
  croppedCanvasContext.drawImage(canvas, left, top, width, height, 0, 0, width, height);

  // These lines can be used to show the cropped part of the image stream that is used
  // to find the code and check if the highlighted area matches the cropped image
  // document.getElementById('croppedSvg').innerHTML = '';
  // const span = document.createElement('span');
  // span.textContent = `${cameraCanvasImageWidth} x ${cameraCanvasImageHeight} | ${videoElementWidth} x ${videoElementHeight}`;
  // span.style.position = 'absolute';
  // span.style.right = `0`;
  // span.style.color = 'white';
  // span.style.display = 'block';
  // document.getElementById('croppedSvg').appendChild(span);
  // croppedCanvas.style.marginTop = '20px';
  // croppedCanvas.style.transform = `scale(1)`;
  // croppedCanvas.style.transformOrigin = `right top`;
  // document.getElementById('croppedSvg').appendChild(croppedCanvas);

  let luminanceSource = new HTMLCanvasElementLuminanceSource(croppedCanvas); // .invert() to support inverted codes

  const hybridBinarizer = new HybridBinarizer(luminanceSource);

  return new BinaryBitmap(hybridBinarizer);
};

There is also a very similar request: #356

Describe the solution you'd like I don't really know how a good solution for this problem would look like in the component interface and also in the library beneath, since it also requires styling of the overlay and matching the displayed overlay with the cropped part.

Maybe it would be a good start to define custom hooks (callbacks) that can be implemented instead of overriding the complete functions.

Additional context

We also have to restart the scanner when the orientation changes in Safari, because the highlighted area does no longer match the croppedCanvas.

  @HostListener('window:orientationchange', ['$event'])
  onOrientationChange(event) {
     ...
  }

odahcam commented 3 years ago

That's very well written indeed, thanks!

I think it might be useful to make the constraints customizable (as input of the component maybe), and also change the default to something like:

I added an experimental input in the component so you can play with that a few weeks ago. But I think it only got published last week. https://github.com/zxing-js/ngx-scanner/blob/42e6ad4f58fd235bd15d688d36050a4ea25fdc01/projects/zxing-scanner/src/lib/zxing-scanner.component.ts#L287-L302

This is just a starting point.

Maybe it would be a good start to define custom hooks (callbacks) that can be implemented instead of overriding the complete functions.

I agree. We're still working on the cropping in the underlying packages, so it was never an easy task to deliever. Now it is looking viable.

One question I have no answer to is: how can we create a customisable mask for croppig without bloating the component?

Maybe we could create customisable component that can be passed as a child to zxing-scanner (eg zxing-scanner-mask) were the scanner component can handle it and adjust/apply only the cropping dimensions to it. But this stills not very clear to me yet.

NilsEngelbach commented 3 years ago

One question I have no answer to is: how can we create a customisable mask for croppig without bloating the component?

How would you define the cropped area in the first place? I think there are two ways:

Define 4 coordinates relative to the image stream, to create an custom rectangle at custom position. E.g. Stream Size 1920x1080, Cropped 400x200 area in center --> Top|Left|Bottom|Right: 640|760|440|1160
Define the size (x|y) of the crop-rectangle either as percentage of the stream size or as absolute values. E.g. 50%|50% or 400|200

Problem with both ways: How to handle different device constraints? On different devices there might be different camera resolutions available.

I like the idea of an optional mask component that can be used as child component, if it would take the same input crop data as the scanner component.

odahcam commented 3 years ago

I like the idea of an optional mask component that can be used as child component, if it would take the same input crop data as the scanner component.

Yes, that's my idea because if we use the same values for cropping and masking it could help "show what the scanner is seeing" and kinda create a visual debugging tool.

I think there are two ways

Yes, but the first way could be a first step and then the second way would calculate the absolute values the first way needs and then would make use of it. So if I had to choose, I'd choose your first way and maybe implement the second way later.

Problem with both ways: How to handle different device constraints?

Yeah, I got myself thinking of that so many times, but it turns out it's not that big of a deal since I already solved similar problems years ago with less technology, so I think it's pretty possible to solve now. Basically we would have to use the zxing-scanner as a 'bounding-box' and work our way from there, something like:

We need to know the zxing-scanner size and make it relative.
Then know the video size.
Then calculate the positioning of the video. It's positioned with CSS now, but as we are already making a lot of calculations and stuff, maybe we should position it with JS (and position: absolute in CSS).
Create the mask using responsive sizes (percentage) relative to the media stream feed. One tricky step here would be to calculate the position and size of the visual cropping rectangle, it's basic math but it does a pretty job fooling us, hehe.
Scale and place the mask absolute positioned over the video feed considering it's scale (eg 1920x1080 scaled down to the webside width). As the mask is responsive, we only need to resize it's frame and align with the video.
Maybe adjust the mask if camera rotation happens?

It looks complex and hard to integrate, but with a child mask component we could direct most of the work to it and just create the positioning logic inside the zxing-scanner. Please consider this is very "on the fly" thinking, so I probably forgot something.

We could also consider improving the cropping tools in the underlying package to see if ends up making this whole issue looks simpler.

ricardotorre commented 3 years ago

I have tried this in a project and it is a massive difference specially for reading pdf417, I hope this lands in the library

shreya-sawardekar commented 3 years ago

@NilsEngelbach could you provide a code snippet on how to use the modifications you have mentioned? I am getting 'BrowserCodeReader' is deprecated.

I want to implement the cropping functionality you have mentioned.

Could you please help me out?

shunta0009 commented 3 years ago

@NilsEngelbach thank you! I am now required to place a focusbox in the center of the ngx-scanner. I saw the code in "Define / Crop the area to search for codes" but I don't know how to handle this code. Can you tell me more specifically?

NexPlex commented 1 year ago

Great package odahcam!

Thank you NilsEngelbach very helpful!!!. Using your code was the only way for me to read the PDF147 with an iPhone.

Everyone else, you need to create a canvas and then play the video on the canvas before calling. BrowserCodeReader.createBinaryBitmapFromCanvas = function (canvas: HTMLCanvasElement): BinaryBitmap.

I also had to wait for the video to load and wait for it to have a size (using setInterval and setTimeout).

Good Luck :)

I did it like this .

in HTML 
   <canvas id="canvas" class="canvas" hidden></canvas>
    <zxing-scanner [torch]="torchEnabled" [device]="deviceCurrent" (deviceChange)="onDeviceChange($event)"
                   (scanSuccess)="onCodeResult($event)"
                   [formats]="formatsEnabled" [tryHarder]="tryHarder" (permissionResponse)="onHasPermission($event)"
                   (camerasFound)="onCamerasFound($event)" (torchCompatible)="onTorchCompatible($event)"
                   (scanError)="onScanError($event)"
                   style="max-height: 60%;   width: auto;  object-fit: contain;"></zxing-scanner>`

scss
 .canvas {
  position: absolute;
  top: 60px;
  left: 20px;
   height: 438px;
   width: 768px;
}

in TS
  ngOnInit(): void {
    this.loadingInterval = setInterval(() => {
      this.loading = this.zXingScannerComponent?.isAutostarting ?? true;
      if (!this.loading) {
        this.croppingBox()
        console.log('this.croppingBox()')
        clearInterval(this.loadingInterval);
      }
    }, 100);
  }

  onCamerasFound(devices: MediaDeviceInfo[]): void {
    this.availableDevices = devices;
    this.hasDevices = Boolean(devices && devices.length);
    this.zXingScannerComponent.getAnyVideoDevice = (): Promise<MediaStream> => {
      return navigator.mediaDevices.getUserMedia({
        audio: false,
        video: {
          width: {min: 640, ideal: 1920},
          height: {min: 400, ideal: 1080},
          aspectRatio: {ideal: 1.7777777778}
        }
      });
    };

    BrowserCodeReader.prototype.decodeFromVideoDevice = async function (
      deviceId: string | undefined,
      previewElem: string | HTMLVideoElement | undefined,
      callbackFn: any,
    ): Promise<IScannerControls> {

      // We had to comment this out because it is a private method...
      // BrowserCodeReader.checkCallbackFnOrThrow(callbackFn);

      let videoConstraints: MediaTrackConstraints;

      if (!deviceId) {
        videoConstraints = {facingMode: 'environment'};
      } else {
        videoConstraints = {
          deviceId: {exact: deviceId},
          width: {min: 640, ideal: 1920},
          height: {min: 400, ideal: 1080},
          aspectRatio: {ideal: 1.7777777778}
        };
      }
      const constraints: MediaStreamConstraints = {video: videoConstraints};
      return await this.decodeFromConstraints(constraints, previewElem, callbackFn);
    }
  }

drawImage(canvas, video, width, height) {
    canvas.getContext('2d', {alpha: false}).drawImage(video, 0, 0, width, height);
  }

  croppingBox() {
    // https://github.com/zxing-js/ngx-scanner/issues/365
    let squareShape = false
    let timeout
    let canvas = <HTMLCanvasElement>document.getElementById("canvas");
    let video = (document.querySelector('zxing-scanner video') as any)
    const fps = 60;
    const width = 1280;
    const height = 720;
    let canvasInterval = null;
    canvasInterval = window.setInterval(() => {
      this.drawImage(canvas, video, width, height);
    }, 1000 / fps);

    video.onpause = function () {
      clearInterval(canvasInterval);
    };
    video.onended = function () {
      clearInterval(canvasInterval);
    };
    video.onplay = function () {
      clearInterval(canvasInterval);
      canvasInterval = window.setInterval(() => {
        this.drawImage(canvas, video, width, height);
      }, 1000 / fps);
    };

    BrowserCodeReader.createBinaryBitmapFromCanvas = function (canvas: HTMLCanvasElement): BinaryBitmap {

      const cameraCanvasImageWidth = canvas.width;
      const cameraCanvasImageHeight = canvas.height;

      let videoElementHeight = (document.querySelector('zxing-scanner video') as any).offsetHeight;
      let videoElementWidth = (document.querySelector('zxing-scanner video') as any).offsetWidth;
      while (videoElementHeight < 1) {
        let tries = 0
        timeout = setTimeout(() => {
          tries++
          videoElementHeight = (document.querySelector('zxing-scanner video') as any).offsetHeight;
          videoElementWidth = (document.querySelector('zxing-scanner video') as any).offsetWidth;
          console.log('tries', tries)
          if (tries > 100) {
            clearTimeout(timeout)
          }
        }, 5000);
      }

      let factor = 1;
      if (videoElementWidth > videoElementHeight) {
        factor = cameraCanvasImageWidth / videoElementWidth;
      } else {
        factor = cameraCanvasImageHeight / videoElementHeight;
      }

      // console.log('factor', factor)
      const width = Math.floor((squareShape ? 250 : 450) * factor);
      const height = Math.floor((squareShape ? 250 : 200) * factor);

      const left = (cameraCanvasImageWidth - width) / 2;
      const top = (cameraCanvasImageHeight - height) / 2;

      const croppedCanvas = document.createElement('canvas');
      croppedCanvas.width = width;
      croppedCanvas.height = height;
      const croppedCanvasContext = croppedCanvas.getContext('2d');
      croppedCanvasContext.rect(0, 0, width, height);
      croppedCanvasContext.fillStyle = 'red';
      croppedCanvasContext.fill();
      console.log('drawImage', canvas, left, top, width, height)
      croppedCanvasContext.drawImage(canvas, left, top, width, height, 0, 0, width, height);

      // These lines can be used to show the cropped part of the image stream that is used
      // to find the code and check if the highlighted area matches the cropped image
      // document.getElementById('croppedSvg').innerHTML = '';
      // const span = document.createElement('span');
      // span.textContent = `${cameraCanvasImageWidth} x ${cameraCanvasImageHeight} | ${videoElementWidth} x ${videoElementHeight}`;
      // span.style.position = 'absolute';
      // span.style.right = `0`;
      // span.style.color = 'black';
      // span.style.display = 'block';
      // document.getElementById('croppedSvg').appendChild(span);
      // croppedCanvas.style.marginTop = '20px';
      // croppedCanvas.style.transform = `scale(1)`;
      // croppedCanvas.style.transformOrigin = `right top`;
      // document.getElementById('croppedSvg').appendChild(croppedCanvas);

      let luminanceSource = new HTMLCanvasElementLuminanceSource(croppedCanvas); // .invert() to support inverted codes

      const hybridBinarizer = new HybridBinarizer(luminanceSource);

      return new BinaryBitmap(hybridBinarizer);
    };

  }

zxing-js / ngx-scanner

Improvements for scanning: choose best resolution and crop the area that should be checked #365

Use the highest camera resolution possible

Define / Crop the area to search for codes