search

SpectacularAI / sdk-examples

Spectacular AI SDK examples
Apache License 2.0
202 stars 35 forks source link

Two questions: How can I access all the data points of the camera's pose positions? Can I load a map? #95

Closed kenhuang1964 closed 9 months ago

kenhuang1964 commented 9 months ago

Question 1: Is there a way I can access the coordinates of all the yellow pose positions? image

Question 2: Can I load a map and localize the camera within the map?

Thanks in advance!

oseiskar commented 9 months ago

1: Yes you can! These are the keyFrames in the Mapping API output. In the replay_to_nerf.py script, they are used like this https://github.com/SpectacularAI/sdk-examples/blob/8ef5ae723a4442a2946bf1ee117ae6a4612d2367/python/mapping/replay_to_nerf.py#L278-L294

They are also stored to disk by the same script, in a Nerfstudio-compatible format and coordinate convention (COLMAP's output format), but if you want to use them for some other purpose, it's probably easiest to modify the replay_to_nerf script where everything is already correctly parsed and accessible in a documented coordinate convention

2: Possible, but currently experimental and not well-documented: You would first have to change the script to output the map in our internal .bin format (i.e., change this line from .csv to .bin, which breaks the rest of the script, but gives you a localizable map), which can then be loaded by the SDK for visual positioning. (Example here)

kenhuang1964 commented 9 months ago

Awesome, thank you so much @oseiskar! I love your enthusiasm :+1:

kenhuang1964 commented 9 months ago

Hey @oseiskar, quick question: can I somehow get the coordinates of the keyframe within the global map? My end goal is to turn it into a sort of graph representation where the positions are the vertices and the edges between them are the distances between two vertices. So, I guess the other question is that is it possible to see if two coordinates are connected (like there is an edge between them)? Also, is it possible to update a loaded map with more coordinates?

oseiskar commented 9 months ago

Key frame (camera) coordinates are a part of the Mapping API output. The sparse co-visibility graph (which KFs share sparse key points) is not available in the current API. In the current tools, the maps cannot be merged or otherwise modified at this stage of the pipeline

kenhuang1964 commented 9 months ago

Thanks!