search

Giodiro / kplanes_nerfstudio

K-Planes model integration for nerfstudio
Other
56 stars 6 forks source link

ns-process-data cannot process dynamic custom data？ #4

Open Kane-LAU opened 1 year ago

Kane-LAU commented 1 year ago

so grateful for your project！And I have test exsiting dnerf data，it worked well. But when i use my own video data （consisting of dynamic object）, it seems working wrong and i don't konw how to process and train data.

Any suggestion about me? Thanks!

LuLing06 commented 1 year ago

The same question here

Giodiro commented 1 year ago

The idea is to add a new data source to nerfstudio. There is the following issue (with little guidance) https://github.com/nerfstudio-project/nerfstudio/issues/2018 but there are also guides to using custom data with nerfstudio: https://docs.nerf.studio/en/latest/quickstart/custom_dataset.html

At least in the beginning, the dataset used is mostly independent from the model used (in this case k-planes). Then you'll have to do some hyperparameter tuning on the configuration of the model to adapt to your dataset.

azzarelli commented 1 year ago

I've had the same issues.

There's a thread in the nerfstudio discord (dated 25/05/23) that says it can't yet process dynamic video data. I don't believe this has been resolved yet. The alternatives discussed are:

  1. Process video using other repos which allow you to create the right transform files
  2. Manually correct the transform.json (from processing video statically) to be train_tranform.json, test_trasnform.json and val_transform.json (i.e. download dnerf data and look at how its setp up compared to static data). This basically means split the dataset into seperate files and add time: n where 0 < n < 1.
  3. Add an issue to this rep

I haven't found any repos which do (1)-> if anyone know any which work please share!

I also haven't had much success with (2) as the frames selected by colmap aren't linearly sequenced with time so requires scrubbing through video and matching each frame to the video. Doing this manually is not really possible, so might be worth scripting this with ffmpeg and/or python?

Any pointers would be much appreciated.

aleatorydialogue commented 1 year ago

So this is where we are for now it seems. No clear way for dynamic video to be converted to data that is compatible with this? Has anyone had any success with including the time element into the data after COLMAP?

aleatorydialogue commented 1 year ago

I've had the same issues.

There's a thread in the nerfstudio discord (dated 25/05/23) that says it can't yet process dynamic video data. I don't believe this has been resolved yet. The alternatives discussed are:

  1. Process video using other repos which allow you to create the right transform files
  2. Manually correct the transform.json (from processing video statically) to be train_tranform.json, test_trasnform.json and val_transform.json (i.e. download dnerf data and look at how its setp up compared to static data). This basically means split the dataset into seperate files and add time: n where 0 < n < 1.
  3. Add an issue to this rep

I haven't found any repos which do (1)-> if anyone know any which work please share!

I also haven't had much success with (2) as the frames selected by colmap aren't linearly sequenced with time so requires scrubbing through video and matching each frame to the video. Doing this manually is not really possible, so might be worth scripting this with ffmpeg and/or python?

Any pointers would be much appreciated.

So I was able to get the output json files from colmap to be in ascending order and added time 0-1 like in the dnerf transforms_train.json. I also added rotation but just set it at 0 for every one since I don't know what they were referring to. But then it says it needs transforms_val.json... Any idea how they generated that? Then there is transforms_train.json as well.

aleatorydialogue commented 1 year ago

I got my data into the correct enough format to run k-planes-dynamic model for training, the color of the scene was present, but never converged into any actual render. Have to know exactly how they produced this format of data

azzarelli commented 1 year ago

So I was also able to create dataset (and it works) by looking at the dnerf parser for nerfstudio. Heres some of the findings:

Any findings from others/comments would be helpful to know if anyone is encounter similar issues?

EDIT: After using the 360_view.py script which comes with the classical Blender datasets I've noticed that they include rotation in their transform files so I can imagine this is where the parameter comes from. I.e. the rotation of a camera around some fixed point, so this could explain why its not an important parameter for Kplanes .

aleatorydialogue commented 1 year ago

So I was also able to create dataset (and it works) by looking at the dnerf parser for nerfstudio. Heres some of the findings:

  • camera_angle_x is importantly used in focal length calculation. I set the param to ~0.7 but it seems to be quite sensitive

  • rotation doesnt seems to affect any of my results, not sure what it does but it doesnt seem to used in the dataparser

  • the train test and val files are a classic data split (For 500 frames I gave 4 to val and 20 to test). However I seemed to get errors when the train file doesnt include all the images. Did anyome else encounter this issue? If so, is there a fix so that I can test images which havent been trained on... maybe its a me problem?

  • finally, you will need to adjust scene scale which is done either in CLI or manually adjusted in code. The nerfstudio implementatiom didnt have a CLI command for this so I manually changed the default scale myself (I rushed through this part so maybe there is a CLI command?)

Any findings from others/comments would be helpful to know if anyone is encounter similar issues?

So you actually generated a dynamic nerf render of your own video? I set my camera_angle_x at 0.69. I set 0 for rotation and time as an even split 0-1 based on how many frames total.

It did load and start training, but never converged. Maybe that is because of the scene scale adjustment you are referencing? I do not know how to change that yet myself, maybe I just wasn't visualizing it correctly...

azzarelli commented 1 year ago

Yes, I was able to do it on several datasets. I beleieve I also used rotation of 0 amd linearly assigned time to images in order as you seem to have done.

Scene scale was important for getting right results but I also adjusted far and near collider planes (which can be done in CLI) and the background colour (random seems best for overall psnr but visually has the worst artifacts).

Honestly, Ive implemented a lot of dynamic models, theyre all pretty challenging to hyperparam tune (and very few offer custom dataset support). They always initially look bad until you hit a sweet spot with the settings, then the results are okay. Theyre definitely not "great" quality so might be worth keeping that in mind too.

Good luck!