search

EdwardLu2018 / wasm-ar

Playing around with Augmented Reality on the Web using OpenCV and WebAssembly
https://edwardlu2018.github.io/wasm-ar/
48 stars 18 forks source link

laggy camera #1

Closed YMZnew closed 3 years ago

YMZnew commented 3 years ago

thank you for sharing your experience . I was testing your code , as the time I click on screen , camera came so laggy with 2FPS . while tracking is well , i think there is sth wrong in detection when click screen or on repeat the detection after lost tracker . would you please help me solve this issue ?

tested on : samsung note 10 plus surface pro 3 camera samsung A50

EdwardLu2018 commented 3 years ago

When you press the screen, it enables detection and tracking, so it will constantly be detecting until it finds a valid detection, then it will start tracking. Detection (which consists of feature point extraction and matching) is slow, even in WASM, so it will definitely lag your camera trying to find the image. I believe what you are seeing is not a bug, but just a desired behavior. You can turn off detection and tracking manually by pressing the screen again. When I get the time, I can implement a more intelligent way to determine when the image is lost and turn off detection automatically.

YMZnew commented 3 years ago

thank you very much for your answer . I've researched for so many methods for detection and their complexity and I think using NFT ( natural featuee tracking ) markers can improve the performance . what do you think about NFT markers?

kalwalt commented 3 years ago

Hi @EdwardLu2018 and @YMZnew i think that fps can be improved a bit using a worker, as we ( me and @ThorstenBux) did in jsartoolkit5 for NFT marker detection and tracking. See threejs_wasm_worker.js and the worker; i was thinking to try this approach also with your code. If you are interested we have founded a new github org for WebAR https://github.com/webarkit

ThorstenBux commented 3 years ago

Nice one, also if you look into artoolkitX.js and the wasm binding there I highly optimised the video hand over . Maybe that is a lag too.

Get Outlook for iOShttps://aka.ms/o0ukef

From: Walter Perdan notifications@github.com Sent: Wednesday, November 4, 2020 8:58:52 AM To: EdwardLu2018/wasm-ar wasm-ar@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [EdwardLu2018/wasm-ar] laggy camera (#1)

Hi @EdwardLu2018https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEdwardLu2018&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341305191%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KFU2SmryruehE%2FGffYf%2FfxHzoJbYyNxudxVAhu5nnhw%3D&reserved=0 and @YMZnewhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FYMZnew&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341315189%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=WS8ixRfk3CML3rAfaeoHD7UcqYRAhuTKX%2FnG3mG0f%2B8%3D&reserved=0 i think that fps can be improved a bit using a worker, as we ( me and @ThorstenBuxhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341335177%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vlsYZJWmvRCeswfDSKjcgJMNzDtp%2FfayyWvAuAWVPqA%3D&reserved=0) did in jsartoolkit5https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341345142%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NlJsPG5pRH166y7UCG9xUR1dQUoe%2FzgflyGoh1oUe4I%3D&reserved=0 for NFT marker detection and tracking. See threejs_wasm_worker.jshttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5%2Fblob%2Fmaster%2Fexamples%2Fnft_improved_worker%2Fthreejs_wasm_worker.js&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341355134%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2Fv9yzmTNvJ6u2tIk%2BW7jDavUX301OtkUzrWKQNyEl6Y%3D&reserved=0 and the workerhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fartoolkitx%2Fjsartoolkit5%2Fblob%2Fmaster%2Fjs%2Fartoolkit.worker.js&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341365130%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5oL5llch%2F%2BwkBMkbq6Z66yo4w0XBlXrAPMO%2FgJfwjlM%3D&reserved=0; i was thinking to try this approach also with your code. If you are interested we have founded a new github org for WebAR https://github.com/webarkithttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwebarkit&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341375124%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=f9GncmN%2BeE6P5ltBFD%2FLWoWVAYcfNqhqYJnHgjBcecc%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FEdwardLu2018%2Fwasm-ar%2Fissues%2F1%23issuecomment-721344739&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341385122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=TlpWOQ8q%2FoK%2BOmdau8lXDD9VwZATX8gEuXa43V1YTks%3D&reserved=0, or unsubscribehttps://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PF32JYM3NTXSJLEMADSOBOHZANCNFSM4S4RLOYQ&data=04%7C01%7C%7C8b9862f20692416de61808d88032e3ed%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637400303341395112%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oanxQusFymiRrM0XRwY0Tl8rYSuAsM529Lmt%2Fy8rHSQ%3D&reserved=0.

kalwalt commented 3 years ago

Yes @ThorstenBux you are right! Here you can find ArtoolkitX.js

EdwardLu2018 commented 3 years ago

Woah! Thanks for all the great comments everyone! Did not expect this project to get this much attention, ha!

@YMZnew I'll be sure to check out NFT tracking! I hear AR.js has an implementation running on the web. Not sure if its running in WASM, though.

@kalwalt I'll definitely try using a web worker when I get the chance. I've used workers in another, similar project I did but never got around to adapting it for this one. I believe I used Comlink in that other project, so I'll probably do the same here eventually.