search

debkbanerji / lego-art-remix

Powerful computer vision assisted Lego mosaic creator · Over 1 million images created (so far!)
https://lego-art-remix.com
GNU General Public License v3.0
288 stars 70 forks source link
computer-vision custom-pictures deep-neural-network javascript lego lego-art lego-art-remix lego-mosaic onnx

readme

Lego-Art-Remix.com

The Intelligent, Easy to Use Lego Mosaic Maker

This project is not affiliated with The Lego Group

Made with ♥ by Deb

Media

These are some articles and videos featuring Lego Art Remix. Note that some were published when the tool was older.

Lego Art Meta Picture

What is it?

In 2020, The Lego Group released the Lego Art theme, which allows people to create a predetermined image using Lego studs.

Lego Art Remix lets you upload your own image, and then uses computer vision to use the studs from a Lego Art set that you already have to recreate the image.

Performance and Security

The computer vision techniques used are pretty inexpensive (with the exception of optional depth map generation), and the resolutions being dealt with are naturally quite low, so as of the time of writing, the algorithm runs quite quickly. This allows for it to be run on the client, and on the machines that I tested, it ran in near real time.

The most computationally expensive part of the process, apart from depth map generation, is generating the instructions, since even pdf generation is done client side.

Since it runs almost entirely within the browser, no image data is sent to a server and so it's very secure. This also makes it much easier for me to maintain and host. The only server code consists of simple increments to anonymously estimate usage, for the purposes for tracking performance in case the static deployment needs to be scaled up, and for the counter in the about section.

Even the deep neural network to compute depth maps is being run entirely within the browser, in a web worker, using a modified version of ONNX.js. I've compiled a version of the library based on this pull request, with a small additional change I made to support the resize operation in v10. The model used is MiDaS - more specifically, the small ONNX version which can be found here.

Citation for model used

Ranftl, René, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer." (2020). IEEE Transactions on Pattern Analysis and Machine Intelligence

Bugs, Feature Requests, and Algorithm Improvements

Direct any concerns or ideas for improvements to the issues tab

As of the time of writing, I don't have all of the sets, and I haven't had much time to test. As a result, there's probably a few bugs, so let me know if you find any.

Algorithm improvement ideas are always welcome. Improvements that maintain the efficiency to within a reasonable degree would allow the algorithm to keep running on the client, which I really like.