Issue with training any NerfStudio methods with the ArkitDataset you provided

[parodiniccolo]

Hi All :) I'm facing some issues when I want to run a NerfStudio Nerf scene with the raw ARKit data you provided.

I'm using the suggested parameter arkit-data to force to load the ArkitScenesDataparser

So the problem is in "get_pose" in the arkitscenes dataparser here, where I do get an empty _framepose.

But by downloading the 3dod data of the same scene directly from ARKitScenes repo, I can successfully train my scene.

[aycatakmaz]

Hi Niccolò,

First of all, thank you very much for your interest in our workshop challenge!

Our download scripts for both tracks do not download all available assets from the ARKitScenes dataset. You are free to use the original dataloader instead in case it suits your needs better for adapting the data for NeRF training.

Regarding the 3DOD data from the dataset, we would like to make the following remarks: 1- Since these are additional GT annotations, we do not allow the 3DOD data for the val/test scenes to be used in our workshop challenge in order to ensure a fair comparison and evaluation across participating teams. So for NeRF training, these GT 3DOD annotations for the test scenes cannot be used. 2- On the other hand, you are free to use any off-the-shelf 3DOD predictor model as long as this predictor was not trained on the val/test scenes of the ARKitScenes on which we evaluate our workshop competition results. So you can use the 3D object boxes predicted from such models while developing your method.

Hope this clarifies your question!

[parodiniccolo]

Thanks for the fast reply.

My goal is to just use the images and the poses extracted from the _lowreswide.traj. As any classic NeRF scene.

Using the 3DOD data (or better the same data you download but "pulled" from the 3dod folder) works for me. Just wondering how come I couldn't be able to train the Nerf scene using "your" data.

[AlexDelitzas]

Hi @parodiniccolo ,

Are you using the lowres or hires frames in your pipeline? We think the issue might be that the dataparser from the ARKitScenes repo will give accurate poses for the lowres frames only. If you are using it to extract poses for the hires frames, you might get an empty frame_pose.

However, the SceneFun3D data parser we provide here can extract accurate poses for both the lowres and hires frames. Specifically, in the latter case it performs rigid body motion interpolation since the timestamps in the lowres_wide.traj are not perfectly synchronized with the hires frames' timestamps. Documentation can be found here and an example script here.

Let us know if that helps!