< img-victor > - Githubissues

In my old blog, the portfolio thumbnail is shown by an animate SVG path like it's drawn by hands. It looks nice and sophisticated.

old-senbazuru old-m

That's not new tech, just combining CSS stroke-dasharray stroke-dashoffset and transform will do the trick. I like the effect and I'd like to use it in 9am.github.io, but the problem is I have to draw all of the thumbnails in paths and you can tell by those old paintings that I'm not good at it.

So for the new site, I find another way to do this. Let's see some demos first:

demo-final

Set goals

Turn \ into SVG \
Easy to use in a web page
Run Fast & Non-blocking experience

Goal1

The \ can be drawn by a canvas and read the raw data by canvas.getContext('2d').getImageData(). It's an array filled in RGBA data of all the pixels of the \. What we need here is a function to turn the imageData into lines, probably in the shape of [[x1, y1, x2, y2], ...]

Task1: function loadImage(url) => imageData
Task2: function translate(imageData) => lines

Goal2

The web component is well supported by browsers now(2022), it's simple to use and the shadow DOM will isolate styles for us. And it works well with popular libs like React or Vue, I think it would be a good choice for this.

Task3: <img-victor src="/img.png"></img-victor>

Goal3

I believe the translate function in Goal1 will be a CPU-bound one. So it would be better if there is a WASM version of it, so we could speed it up.

Task4: WebAssembly

We don't want the translation process to hang the page since it might take some time to finish. So it's better to put it in a web worker.

Task5: Web Worker

OK, Let's build this step by step.

Step1: Build a Web Component skeleton

We'll build a simple Web Component called \, which simplely works like a \, but rendering the image with a canvas.

const template = document.createElement("template");
template.innerHTML = `
    <style>
        :host {
            display: inline-block;
        }
    </style>
    <canvas id="canvas"></canvas>
`;

class ImageVictor extends HTMLElement {
    static get observedAttributes() {
        return ["src"];
    }
    static async loadImage(url) {
        // TBD task1
    }

    constructor() {
        super();
        this.attachShadow({ mode: "open" });
        this.shadowRoot.appendChild(
            template.content.cloneNode(true)
        );
    }

    async attributeChangedCallback(name, prev, next) {
        if (prev === next) {
            return;
        }
        switch (name) {
            case "src": {
                const img = await ImageVictor.loadImage(this.src);
                this._render(img);
                break;
            }
            default:
                break;
        }
    }

    _render(img) {
        // TBD task2
    }

    get src() {
        return this.getAttribute("src");
    }

    set src(val = "") {
        this.setAttribute("src", val);
    }

    connectedCallback() {
        if (!this.hasAttribute("src")) {
            this.src = "";
        }
    }

    disconnectedCallback() {}
}

// task3
window.customElements.define("img-victor", ImageVictor);

The Web Component attaches the predefined template in the shadow DOM. It watches the 'src' attributes, fetching the image with a static function loadImage after 'src' is updated, then _render the canvas with the imageData.Finally, it's registered by the name 'img-victor'. (Notice: The getImageData follows CORS, so don't try to load the image from another origin, there is no way to pass it unless the image server allow your javascript origin)

Now we implement the loadImage and _render function.

For the loadImage, we can get the ImageData with an Image and a Canvas.

static async loadImage(url) {
    return new Promise((resolve, reject) => {
        let img = new Image();
        img.crossOrigin = "anonymous";
        img.onload = () => {
            try {
                const canvas = document.createElement("canvas");
                const ctx = canvas.getContext("2d");
                const { width, height } = img;
                canvas.width = width;
                canvas.height = height;
                ctx.drawImage(img, 0, 0, width, height);
                resolve(ctx.getImageData(0, 0, width, height));
            } catch (error) {
                reject(error);
            }
        };
        img.onerror = (error) => reject(error);
        img.src = url;
    });
}

For the _render, it renders the ImageData we just got from loadImage to the canvas in the shadow DOM.

_render(img) {
    const canvas = this.shadowRoot.querySelector("#canvas");
    const ctx = canvas.getContext("2d");
    canvas.width = img.width;
    canvas.height = img.height;
    ctx.putImageData(img, 0, 0);
}

Put it together:

Step2: Translate ImageData to line vectors

This is the hard part. The first idea that comes to my mind is that I need some kind of line detection algorithm. After some searching, Hough Transform and Canny edge detector pop out. But the Canny edge detector returns points instead of line segments, and the number of points is hudge when the image gets complicated. Joining those points into lines is a difficult job. Then I found this algorithm LSD: a Line Segment Detector . It's fast and the output looks promising, and there is even a C version implementation on the website.

Compile C to WASM

I compile the code with emscripten. It generates some glue codes in javascript and a separate wasm file. Notice that although emscripten proxy the C function to javascript for us (Module._lsd), we have to manage the memory ourselves (Module._malloc Module._free). So it takes a pre.js to bridge the C function properly.

Module['gray'] = ({ data, width, height }) => {
    const arr = new Float64Array(width * height);
    for (let i = 0; i < data.length; i += 4) {
        arr[i/4] = data[i];
    }
    return arr;
};
Module['chunk'] = (input = [], size = 7) => {
    const result = [];
    const len = input.length;
    for (let i = 0; i < len; i += size) {
        result.push(input.slice(i, i + size));
    }
    return result;
};
Module['translate'] = ({ data, width, height }) => {
    let lines;
    let resultPtr;
    let countPtr = Module._malloc(8);
    const dataPtr = Module._malloc(width * height * 8);
    const grayData = Module.gray({ data, width, height});

    try {
        Module.HEAPF64.set(grayData, dataPtr / 8);
        resultPtr = Module._lsd(countPtr, dataPtr, width, height);
        const len = Module.getValue(countPtr, 'i32');
        lines = Module.chunk(
            Module.HEAPF64.subarray(resultPtr / 8, resultPtr / 8 + len * 7),
        );
    } finally {
        Module._free(dataPtr);
        Module._free(countPtr);
        if(resultPtr) {
            Module._free(resultPtr);
        }
    }
    return lines;
};

Once we have the bridge ready, compile the wasm in CLI, and specify the glue output as ESM.

emcc lsd.c -o lsd.mjs -sEXPORTED_FUNCTIONS="['_malloc', '_free', '_lsd']" -sALLOW_MEMORY_GROWTH=1 --pre-js pre.js

Put it all together, now we have the function translate(imageData) => lines.

import { ImageVictor } from "/js/component.js";
import Module from "/js/lsd.mjs";

// The default export of lsd.mjs is a function returns a Promise,
// which will be resolved to an object with all the bridge functions.
const LSD = await Module();
const img = await ImageVictor.loadImage("/img/senbazuru.png");
const lines = LSD.translate(img);
// lines: [start, end] [[x1, y1], [x2, y2]]
[
  {
    "0": 168.9860607982964, // x1
    "1": 180.61778577533605, // y1
    "2": 169.45004281672658, // x2
    "3": 155.6032063493381, // y2
    "4": 3.9579894109323166,
    "5": 0.125,
    "6": 35.104631104903746
  },
  ....
]

(Notice: The LSD in C only takes gray data of the image, it can be done by applying a filter: grayscale(1) to the canvas in loadImage, let the browser do the job for us)

Step3: Rewrite the `_render` function

Since we have the lines, we can render the SVG path to the Web Component.

Put the SVG and path in the compnent template.

template.innerHTML = `
    <style>
        ...
        path {
            fill: none;
            stroke: black;
            stroke-width: 0.2%;
        }
    </style>
    ...
    <svg id="svg" xmlns="http://www.w3.org/2000/svg" width="100%" height="100%" role="img" aria-labelledby="title">
        <title id="title"></title>
        <path id="path" />
    </svg>
`;

Update 'g' of \ after lsd

    async _render(img) {
        ...
        // render path
        const { translate } = await LSD;
        const lines = translate(img);
        const path = this.shadowRoot.querySelector("#path");
        const d = lines
            .map(([x1, y1, x2, y2]) => `M${x1},${y1} L${x2},${y2}`)
            .join(" ");
        path.setAttribute("d", d);
        // set svg viewBox
        const svg = this.shadowRoot.querySelector("#svg");
        svg.setAttribute("viewBox", `0 0 ${img.width} ${img.height}`);
    }

Make it move using the old trick

    draw(path, lines) {
        const len = Math.max.apply(
            null,
            lines.map(([x1, y1, x2, y2]) => Math.hypot(x2 - x1, y2 - y1))
        );
        path.style.strokeDasharray = len;
        path.animate([{ strokeDashoffset: len }, { strokeDashoffset: 0 }], {
            duration: 5000
        });
    }

Now we can remove the temp canvas in the Web Component. It's done.
But wait... did we forget something? I'll show you the problem.

Step4: Web Worker

The LSD takes a lot of calculation, and since javascript works 'single-thread', when LSD is working, it blocks everything on the page. As you can see in the example, the user couldn't input until LSD finishing.

Let's involve Web Worker to make LSD work in a separate thread.

worker.js

// translate in worker
import Module from "/js/lsd.mjs"
const LSD = Module();

self.onmessage = async (evt) => {
    const { translate } = await LSD;
    const lines = translate(evt.data);
    postMessage(lines);
};

component.js

    _translateWithWorker(data) {
        return new Promise((resolve, reject) => {
            const worker = new Worker("/js/worker.js", { type: "module" });
            worker.onmessage = (result) => resolve(result.data);
            worker.onerror = (err) => resolve(err);
            worker.postMessage(data);
        });
    }

(Notice: { type: "module" } for worker only works in Chrome 2022/3. We could rebuild the glue code as worker directly to run this in other browsers)

Now let's check out the new version.

Not only the blocking is gone, but also the whole render process is speeding up because every img-victor runs in parallel. Finally, we got all goals filled.

The lib is available

NPM npm install @9am/img-victor
SKYPACK https://cdn.skypack.dev/@9am/img-victor
GITHUB

Features supported:

Loading Indicator
Workers Pool
Worker Plugin
Path Stroke
TagName

References

LSD: a Line Segment Detector

image.LineSegmentDetector - Line Segment Detector for images

The LSD: Line Segment Detector algorithm compiled to javascript with emscripten

@9am 🕘

Read more articles at 9am.github.io

Find other things I built on GitHub and NPM

Contact me via email

9am / 9am.github.io

< img-victor > #3

Set goals

Step1: Build a Web Component skeleton

Step2: Translate ImageData to line vectors

Compile C to WASM

Step3: Rewrite the `_render` function

Step4: Web Worker

The lib is available

@9am 🕘

9am / 9am.github.io

< img-victor > #3

Set goals

Step1: Build a Web Component skeleton

Step2: Translate ImageData to line vectors

Compile C to WASM

Step3: Rewrite the _render function

Step4: Web Worker

The lib is available

@9am 🕘

Step3: Rewrite the `_render` function