NFT (Natural Feature Tracking) on AR.js

[nicolocarpignoli]

AR.js is currently able to recognize markers based on artoolkit marker, so the ones with the dark (black) border around shapes.

NFT (Natural Feature Tracking) can bring new life to AR.js, we eventually will be able to scan and recognize also 2D images, for example.

There is a branch for NFT on artoolkit, the library that AR.js is using for recognization. @kalwalt is working on his fork of jsartoolkit (https://github.com/kalwalt/jsartoolkit5/pull/1) in order to bring the NFT branch alive again, including examples. They are outdated, but some times ago they where working.

So: if we can help him/support him bringing the NFT branch alive and working again, we can then think of use that jsartoolkit version in order to bring NFT feature to AR.js.

Just wanted to make this more visible. If someone is willing to help and has experience about that, please let us know :)

[kalwalt]

Thank you @nicolocarpignoli we need to take this beast by the horns!

[nicolocarpignoli]

Update: @kalwalt has been able to make NFT branch working again on jsartoolkit!

Now we need to find a way to:

• create NFT markers starting from a 2D image
• possibly load such markers dynamically on jsartoolkit example
• replace/port the NFT jsartoolkit version into AR.js

[kalwalt]

Thanks for the enthusiasm @nicolocarpignoli !! :1st_place_medal:

create NFT markers starting from a 2D image

i want to look to this! i want to try with my own image....

[kalwalt]

@nicolocarpignoli Idea: i am working on adding NFT to jsartoolkit5 https://github.com/kalwalt/jsartoolkit5/pull/1 , that is all is together (marker, multimarker, barcode, and NFT) but maybe one person could be interested to only the markerless NFT tech, so i was wondering that it will be cool to develop a lighter version of jsartoolkit with only NFT, what do you think?

[nicolocarpignoli]

In my opinion it could be interesting, but I will see that as a 'nice-to-have'. Priority should to simply (lol) bring NFT to AR.js.

[kalwalt]

Yes of course! 😀

[nicolocarpignoli]

Things are moving! @Carnaux has started a repo to bring to the web a tool to create NFT marker! https://github.com/kalwalt/jsartoolkit5/pull/1#issuecomment-514449013

[kalwalt]

For anyone interested in this topic: i uploaded some examples in my github webpage kalwalt.github.io/kalwalt-interactivity-AR/ to help testing on mobile devices. For now i uploaded only the threaded version of jsartoolkit5 with NFT, before using it read carefully the infos especially the Spectre issue!! I will upload the other examples ( the non threaded version) when i will have a bit of time, sadly i have not so much in this last period...

[MiguelNunes71]

Congratulations kalwalt, i successfully tested your NTF examples kalwalt.github.io/kalwalt-interactivity-AR/ on both Android and IOS devices !

A question, how can we detect when the marker is visible and when it is hidden, so we tell to start or to stop other actions like animation and sound?

[kalwalt]

Congratulations kalwalt, i successfully tested your NTF examples kalwalt.github.io/kalwalt-interactivity-AR/ on both Android and IOS devices !

A question, how can we detect when the marker is visible and when it is hidden, so we tell to start or to stop other actions like animation and sound?

You should look in the console you should receive some notifications. Not sure if at the moment it is implemented a event listeners for this, Sorry! but i will look into it... BTW could you be more specifc with performances and devices? Thank you! :smile:

[kalwalt]

We @kalwalt @nicolocarpignoli @ThorstenBux are on the road to merge the jsartoolkit5 NFT feature into the master branch of artoolkitx/jsartoolkit5 in this PR. :smile: :+1:

This means that in another step will be integrated into AR.js !! :raised_hands: :tada: If you want to test the examples go to this page kalwalt.github.io/jsartoolkit5/examples/ and report your tests and issues in the artoolkitx/jsartoolkit5 PR or in the PR where i started this hard/crazy work.

See you soon for nice updates !! :100: :bomb: :fire:

[TheBricktop]

holy moses, this is the most profound work on AR,js and AR in general ive seen in years, if this would be implemented the pain of cross compiling android and ios apps would be gone forever

[kalwalt]

holy moses, this is the most profound work on AR,js and AR in general ive seen in years, if this would be implemented the pain of cross compiling android and ios apps would be gone forever

Yes @TheBricktop this is real! We hope to merge into the artoolkitx/jsartoolkit5 master branch in january and after include it in AR.js If you find some issues please open it at https://github.com/kalwalt/jsartoolkit5/issues or leave a comment in the PR

[TheBricktop]

Will definitely try it out

[cvasquez-github]

Thank you @nicolocarpignoli and @kalwalt for this huge effort! Please let me know if can contribute in any way. Thanks!

[kalwalt]

The NFT feature finally has been merged in the artoolkitx/jsartoolkit5 master branch !!! It's an EPIC MOMENT :smile: :tada: :bangbang: :bomb: We, me and @nicolocarpignoli , are working on the Ar.js side to integrate the new jsartoolkit5 libs with NFT support into the core. You can follow our progresses in this PR https://github.com/jeromeetienne/AR.js/pull/705 many thanks to all the people involved in this wonderful process, not to help for coding but also for testing and for the support. I am very happy of this community and of opensourcing. I have also written this article on my blog for the historic event: https://www.kalwaltart.com/blog/2020/01/21/nft-natural-feature-tracking-with-jsartoolkit5/ and please spread the word! Thank you all!

[cvasquez-github]

OMG!!! Congratulations!!! A huge contribution to the AR Development ecosystem!!! Thank you so much for reaching this milestone! Which example should we consider as a base for testing it main_threejs_wasm_worker.html? Regarding NFT Marker creation, would it work properly with fset files generated with https://carnaux.github.io/NFT-Marker-Creator/ ? Thanks!

[kalwalt]

HI @cvasquez-github! Thank you! Yes consider to test that example and if you want create yours own NFT markers you can use the https://carnaux.github.io/NFT-Marker-Creator/ web version or the command line app https://github.com/Carnaux/NFT-Marker-Creator enjoy!

[cvasquez-github]

Awesome, thanks @kalwalt ! BTW which camera calibration parameters dat files should we use? I see the example threejs_wasm_worker uses _camerapara-iPhone 5 rear 640x480 1.0m.dat file.

[linkadjausal]

Marvelous. Thank you very much for contributing a very important feature of augmented reality, in JavaScript !!

[franbrennan]

Simply... THE... BEST... NEWS... 😎👏👏👏👏👏❤️ Huge thanks for all the amazing work that's been done!

[markerio]

That’s perfect guys, thanks a million!

[nicolocarpignoli]

closed, see related PRs.

[abhishekQ]

can we play an mp3 file with nft marker

[subjectdenied]

@abhishekQ : the markers are only used for AR features, audio is done at the "3d-engine" level (three-js), so it should work. i did this successfully using arNFT (what ar.js is based on afaik) and three-js

[abhishekQ]

@abhishekQ : the markers are only used for AR features, audio is done at the "3d-engine" level (three-js), so it should work. i did this successfully using arNFT (what ar.js is based on afaik) and three-js

I want to play sound with 3d model when nft marker is detected. if you have code can you please share it with me

[subjectdenied]

https://www.trickymind.at/ar get the tracking-image by clicking the button on the bottom-right at the screen