search

jc211 / NeRFCapture

An iOS app that collects/streams posed images for NeRFs using ARKit
MIT License
246 stars 27 forks source link

How to use captured depth images? #10

Open jgnooo opened 1 year ago

jgnooo commented 1 year ago

Hello, Thank you for the great work!!

I'm using NeRFCapture app for obtaining RGB-D images and poses. However, when generating point cloud using these data, It is something weird.

image

I think it is because of depth image scale. How to resolve it, and What is the scale of the depth map from the NeRFCapture?

Thank you 🥹

jc211 commented 1 year ago

The depth image is upscaled quite a bit. I think its something in the order of 250x120 (not sure what the exact number is). I think what you are seeing is mostly the interpolation.

garylidd commented 1 year ago

Hi, I have a problem converting the captured depth image to real depth. The depth map is 3 times larger than the color image and I don't know how to extract the true depth correctly. I have attached relevant images for your reference, can you help me with that ? ref.zip

jc211 commented 1 year ago

Looks like the offline depth is not working properly. I'll try and fix it soon. Sorry about that.

oseiskar commented 11 months ago

As a work-around, we (Spectacular AI), have released an alternative offline-focused iOS data collection app that can record both depth and high-resolution video at high FPS. Its outputs can be used for InstantNGP, Nerfstudio (step-by-step tutorial) and probably even SplaTAM (see here).

pablovela5620 commented 11 months ago

I would also recommend looking into polycam app, they enable a developer mode that allows for getting rgb + depth + camera poses/intrinsic parameters.

https://github.com/PolyCam/polyform

this may simplify the need for having to use a network connection

nitthilan commented 10 months ago

Any update on the fix for the offline version?

jc211 commented 9 months ago

Hey sorry I have not had the chance to look at this yet. However, if anyone is compiling from source and would like to have a go at fixing it, these are the lines that need changing:

https://github.com/jc211/NeRFCapture/blob/312cb01efd5bd84e30cf9dea32a3a12a70abbc5b/NeRFCapture/DatasetWriter.swift#L173C1-L176C18

The problem is that depthBuffer!.pngData() does not work and should be replaced by a manual operation that converts the depth buffer into a 16 bit number that can be written to a png.

Zhangyangrui916 commented 8 months ago

I'm not familiar with processing pixels on IOS. So I save depth as binary and process it on PC with Python. https://github.com/Zhangyangrui916/NeRFCapture/commit/3707209eff4da752469c12a58330b38543ad6055