search

AR-js-org / AR.js

Image tracking, Location Based AR, Marker tracking. All on the Web.
MIT License
5.47k stars 932 forks source link

Enhancing location-based AR with computer vision (inviting a discussion more than anything else) #199

Open nickw1 opened 3 years ago

nickw1 commented 3 years ago

Do you want to request a feature or report a bug?

Feature

What is the current behavior?

Location-based AR works nicely, but it would be really good if, at some point in the future, objects could be placed accurately as if they were on the ground.

If the current behavior is a bug, please provide the steps to reproduce.

Please mention other relevant information such as the browser version, Operating System and Device Name

Any

What is the expected behavior?

N/A

If this is a feature request, what is motivation or use case for changing the behavior?

This is really just to invite discussion rather than make a feature request as such. What would be really nice in the future for location-based AR is to allow objects to be placed accurately on the ground to improve the realism - see for example here; it could be significantly improved by placing the augmented content on the ground. It looks like - in theory - this could be done using OpenCV using algorithms to detect the ground plane. We would however need to use OpenCV.js from JavaScript.

Is there anyone here who has had OpenCV, and particularly OpenCV.js, experience who could maybe comment on the feasibility or otherwise of this? From doing a bit of preliminary research it looks possible on native but not sure how feasible it would be on the web platform.

JamesBotterill commented 3 years ago

This is obviously the missing link for AR.js is to have a SLAM based implementation in general, this is part of the webXR spec with the Hit Test API but until Apple take notice of it is dead in the water.

I think it's mentioned in this thread here: https://github.com/jeromeetienne/AR.js/issues/227

albjeremias commented 3 years ago

@JamesBotterill as said on that thread jsartoolkit already implements that in a very efficient way (artoolkit is based on opencv). I think that a better approach is to pickup the old artoolkit code and modify for this feature which is about the ground for location-based AR not for NFT tracking.

Anyway I wonder if opencvjs would be the best fit... It would be possible to do it, but it's quite complicated on how-to compute a matrix which was previous computed by jsartoolkit... And would be better to have an explanation on how location-based AR works in ARjs

nickw1 commented 3 years ago

@JamesBotterill it looks like the link discusses NFT, which is obviously already present. Is NFT only for detecting well-defined images, or could it be extended to recognise something as (often) vague as the ground, or would it only work with well-defined features such as lone trees, or gates, for instance?

Could jsartoolkit potentially allow ground plane detection? I mentioned OpenCV because I assumed (perhaps wrongly) that jsartoolkit could not do this.

Location-based AR in AR.js is relatively simple; it uses the geolocation API to get your position plus the JavaScript sensor APIs - within A-Frame - to get the current device orientation. The sensor work is largely done for us from within A-Frame, or more specifically three.js's DeviceOrientationControls which is used by A-Frame.

So it would be - presumably - a case of finding the ground plane, and then adjusting the position given by location-based AR so that it's fixed to the nearest point on the ground plane.

Sorry if there are a lot of questions but I'm keen to work on this and it would be good to get input from those who have prior experience in the NFT/computer vision side of things.

nicolocarpignoli commented 3 years ago

Maybe @ThorstenBux can help on this? Some hints?