[ML-Kit] append a DOM element with matrix3d transform for each scanned

[JanMisker]

I'm testing with the ml-kit branch, works really well already! I see that the coordinates of each recognised element are also included, it would be cool to put a 'box' around each. You could of course simply draw lines on a canvas or so, but it could also be possible to add a DOM element that is precisely position on the right coordinates, and also has a matrix3d transform applied such that anything inside that element gets the correct perspective. In my opinion this could be added in the plugin, in the web layer.

This SO post has a lot of info as well as example implementations. https://math.stackexchange.com/questions/296794/finding-the-transform-matrix-from-4-projected-points-with-javascript

I'm putting this here as reference also for myself, I would be willing to help out with this feature.

[thegnuu]

@JanMisker I was already testing this ;)

Unfortunately, I had some issues I could not resolve until now. So my idea was to fix the other issues and add this as a feature in a future release ;)

My idea was to return the coordinates and render this stuff in the DOM as you suggested, I was partially successful doing this. My main problem was correctly mapping the points returned from ML-Kit to pixel-related values that match the DOM. It worked in some cases, but somehow I had weird issues with the orientation, the points were not always returned from the top left corner as I expected. I was just working on iOS, maybe it would be easier to start on Android, but I guess it will be more or less the same problem there...

[JanMisker]

Coolcool, not the most important feature I'd say but nice to have.

[Jopie01]

For Android there is already code for this, the files BarcodeGraphic.java and GraphicOverlay.java implements that. It's taken from the MLKit (https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart/app/src/main/java/com/google/mlkit/vision/demo/java/barcodescanner) example app, but unfortunately it's not hooked up to the scanner it seems to be lost or forgotten.

[JanMisker]

Nice, but I think it makes more sense to implement it on the HTML/JS side of the plugin. That way developers can just attach something to the DOM elements that are provided to them. Maybe just a simple ::after { position: absolute; ... } css declaration to put a border.

[Jopie01]

I think it makes more sense to implement it on the HTML/JS side of the plugin.

Agree on that, but the example code can help to get the coordinates right. When you search the internet, you will find that it will be a pain to get it right because of the different screens, camera's and image sizes. Also it would be nice to have the code also be able to do the reverse. Sending a set of coordinates to draw a rectangle in which the barcode should be in order to get a 'proper' result. The scanner can have already decoded the barcode but the result is still invalid if the barcode is not in that rectangle.

[peitschie]

Just noting, I've been able to do this very trivially with a full-screen canvas overlay element (with a slight bug on iOS, #217)

The basic code is something like:

// Create a full-screen canvas overlay
const canvas = document.createElement("canvas");
canvas.style.position = "fixed";
canvas.style.top = "0";
canvas.style.left = "0";
canvas.width = screen.width;
canvas.height = screen.height;

// Append to document
document.body.appendChild(canvas);
const ctx = canvas.getContext("2d");
// Fix scaling, as per #217 (on iOS at least...)
ctx.scale(0.7, 0.7);

// Convenience function to draw the corner points as a path
function drawOverlay(points) {
  ctx.fillStyle = "orange";
  ctx.strokeStyle = "orange";
  ctx.beginPath();
  ctx.moveTo(...points[0]);
  for (const [x, y] of points.slice(1))
    ctx.lineTo(x, y);
  ctx.lineTo(...points[0]);
  ctx.stroke();
}

// Start barcode scanning
BarcodeScanner.start({}, (result, err) => {
  if (err) return; // TODO handle error cases

  // Draw the overlay on screen
  drawOverlay(result.cornerPoints);
})

Just leaving it here in case it's helpful for someone else 🙂

[thegnuu]

This looks promising! Thank you for the input @peitschie