Questions about preprocessing the data

[fyhdev]

Hi, nice work! Really impressed by it. However, I still got some questions: a) When I download the Nerf-on-go dataset(eg:fountain), I get three files: checkpoints,predictions and tensorboards. Could you explain the meaning of each file? b) In the rendering process, it is said that we need the checkpoints and the trajectory. So is this checkpoint from the training output and the trajectory from the interactive viewer? If so, how can I export the trajectory as a file in my repository after I can visualizing the interactive viewer in the viser? c) In the training process, if I want to train based on the Nerf-on-go dataset, where is the data path. Is the data path the path of checkpoints, predictions or tensorboards? These are several questions confusing me. I will be really grateful if you could answer my questions!

[jkulhanek]

Hi, a) Where did you download it from? What you are describing are the trained checkpoints, not the datasets. Perhaps you downloaded the checkpoints instead of the dataset? The dataset should be here: https://huggingface.co/datasets/jkulhanek/nerfonthego-wg/tree/main b) You are correct. If you open the viewer, in the render tab you can create the trajectory and by pressing the 'export trajectory' button you save the trajectory as a file. c) Please follow the link to the data in the README. The data is hosted on huggingface. It is different from the data path of checkpoint/predictions/tensorboard.

Best, Jonas

[fyhdev]

Thanks for you reply! It helps me a lot. But now I found a new problem that when I am rendering a video, my terminal said that "nerfbaselines render has no such option --trajectory". Is it because my nerfbaselines=1.01 and I should have a greater version of it?

[jkulhanek]

I see, please use the ‘nerfbaselines render-trajectory’ command for your version

[fyhdev]

Thanks, so checkpoints, predictions and tensorboards are results trained from the images right? By the way, get another problem in rendering the video. This is the screenshot of my error. It is said that need at least one array to stack. Can't figure out why?

[jkulhanek]

Please always attach full command and full traceback of the error

[fyhdev]

Got it. Sorry for that. This is the screenshot of my whole error. It seems like an error caused by my environment. ![Uploading Screenshot 2024-07-27 131503.png…]()

[fyhdev]

[jkulhanek]

Can you please also share the trajectory file? Seems like you have 0 frames in your trajectory? Did you add at least two keyframes to the trajectory in the viewer?

[fyhdev]

Yes, you are right. I didn't add keyframes. How can I render a video by the way? Using the command you gave me 'nerfbaselines render-trajectory’, I can only get several images on a single view. How can I get a video on a 360 view? Thanks for your reply!

[jkulhanek]

When you play the trajectory in the viewer it is a video or does it jump between keyfranes? Perhaps you change the interpolation type to none under render setting in the viewer. Also note, you can render the trajectory directly from the viewer, no need to run the render-trajectory command

[fyhdev]

I see. It jumps between keyframes. I did find a render button in the viewer. However, there will be an error when I press that button.

[fyhdev]

Also, the type of my interpolation is kochanek-bartels. Is it ok?

[jkulhanek]

Kochanek-bartels is ok. Can you share the trajectory.json file?

[fyhdev]

trajectory-NEW.json This is my trajectory file

[fyhdev]

Also, where can I get the original image on the photo tourism dataset. I can only get the trained data from your link.

[jkulhanek]

Your trajectory contains several keyframes with a single camera view. You have to add a key frame, move the camera somewhere, add new keyframe, etc... to build a trajectory. Also, the reason ffmpeg is not working for you is probably an old version of ffmpeg not matching your mediapy version. Perhaps try updating ffmpeg.

[abenbihi]

Hi,

Thanks for releasing the code and the documentation. I am running the code on a custom COLMAP dataset

When rendering new trajectories, it seems that the render-trajectory command of nerfbaselines expect the checkpoint folder to hold a file nb-info.json with information on the background and the expected scene scale, which I assume should be generated by the code. However, there is no such folder in the checkpoint directory.

Any idea what may be going on here?

I am running the latest version of the main branch of wild-gaussians with nerbaselines==1.0.2

Below is the stack trace

Thanks

info: Loading checkpoint checkpoint-70000/
Traceback (most recent call last):
  File "conda/envs/wg/bin/nerfbaselines", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/click/core.py", line 1157, in __call__                                                                                                
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/click/core.py", line 1078, in main                                                                                                    
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/click/core.py", line 1688, in invoke                                                                                                  
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/click/core.py", line 1434, in invoke                                                                                                  
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/click/core.py", line 783, in invoke                                                                                                   
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/nerfbaselines/utils.py", line 341, in wrapped                                                                                         
    raise e
  File "conda/envs/wg/lib/python3.11/site-packages/nerfbaselines/utils.py", line 334, in wrapped                                                                                         
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "conda/envs/wg/lib/python3.11/site-packages/nerfbaselines/cli/render.py", line 120, in render_trajectory_command                                                                  
    assert (checkpoint_path / "nb-info.json").exists(), f"checkpoint path {checkpoint} does not contain nb-info.json"                                                                                             
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: checkpoint path checkpoint-70000/ does not contain nb-info.json

[jkulhanek]

Sorry for the issue, I'm currently working on an update that should fix the issue. I assume you train the model using the train.py script, not nerfbaselines train, but now you want to use the checkpoint with nerfbaselines * command is that correct? In that case, can you just add an nb-info.json file to the checkpoint directory with the following contents: {"method": "wild-gaussians"}? Does it (temporarily) resolve the issue?

[abenbihi]

No worries and thanks for the quick reply!

Yes, manually adding the nb-info.json solves the issue.

Indeed, I trained the model using the wild-gaussians train.py and wanted to then render views from the trained model. The documentation suggested to use nerfbaselines for rendering but if there is a wild-gaussian rendering script that is independent of nerfbaselines, that would work too.

[jkulhanek]

This should now be resolved: the train.py script will generate nb-info.json