Video rendering - kplanes-dynamic (AssertionError: Initialized model with time-planes, but no time data is given)

[Anji-Builds]

I have trained the kplanes-dynamic model with the proposed dnerf synthetic dynamic dataset. However, as there is no clear instruction on how to render the video for the same, I am struggling to render video result. I have few questions regarding the process.

1. How to render videos for kplanes-dynamic model?
2. Is camera path necessary or could I directly use interpolate option?
3. Do we need to use the viewer and do something in order to start rendering successful? or is it possible directly to render with using viewer interface.
4. How does the Time slider in viewer affect the rendering?

I had come across the following error while attempting to render.

The following are the steps that led to the error.

1. trained for dnerf sythetic dynamic scene --> jumpingjacks
2. ns-render interpolate --load-config outputs/jumpingjacks/kplanes-dynamic/2024-09-05_115237/config.yml --output-path renders/jumpingjacks/jj_interpolate.mp4

Could anyone guide me with the steps to attain the rendering. Thanks in advance.

[Tavish9]

I have trained the kplanes-dynamic model with the proposed dnerf synthetic dynamic dataset. However, as there is no clear instruction on how to render the video for the same, I am struggling to render video result. I have few questions regarding the process.

1. How to render videos for kplanes-dynamic model?
2. Is camera path necessary or could I directly use interpolate option?
3. Do we need to use the viewer and do something in order to start rendering successful? or is it possible directly to render with using viewer interface.
4. How does the Time slider in viewer affect the rendering?

I had come across the following error while attempting to render.

The following are the steps that led to the error.

1. trained for dnerf sythetic dynamic scene --> jumpingjacks
2. ns-render interpolate --load-config outputs/jumpingjacks/kplanes-dynamic/2024-09-05_115237/config.yml --output-path renders/jumpingjacks/jj_interpolate.mp4

Could anyone guide me with the steps to attain the rendering. Thanks in advance.

Hi, I'm glad to help.

Let me address your questions one by one:

1. How to render videos for kplanes-dynamic model? To render, you can use the command ns-render --help to check the usage instructions. Note that from version v1.0.0 onwards, the render type supports dataset. So if you'd like to render the same image as in the eval dataset, upgrade your nerfstudio version first.
2. Is camera path necessary or could I directly use interpolate option? It depends. If you want to render along an arbitrary path in the scene, use camera-path. If you're following the train/eval path but with different views, interpolate is the way to go. If you prefer rendering the same views as in the train/eval dataset, choose dataset.
3. Do we need to use the viewer and do something in order to start rendering successful? or is it possible directly to render with using viewer interface? Rendering is performed via ns-render. Unless you're customizing camera paths, viewer isn't necessary for rendering.
4. How does the Time slider in viewer affect the rendering? The time slider in the viewer controller only impacts the output you see in the viewer interface, it does not affect the rendering result.

The error you encountered is caused by the interpolate. Since kplanes-dynamic requires time in every frame, but the interpolate mode only interpolates camera poses, excluding time, as a result, this leads to the error you're seeing.

To resolve this, you can either interpolate time yourself, or just use dataset mode to follow the pre-defined path. This should solve your issue.

[Anji-Builds]

@Tavish9 Thank you for your response. Really appreciate it.

However, I still have doubts. I am unable to still understand how to obtain/render a time variant kplanes-dynamic video. I too have obtained the images but it would be nice to get a video.

I cannot find documentations for the video generation of kplanes-dynamic.

I was able to generate video for static scene using --interpolate only. How else do you suggest I generate a video? Direct ns-render with load-config does not generate any video for me. The other sub commands are {camera-path,interpolate,spiral,dataset} and thats how I decided to go for interpolate choice.

[Tavish9]

@Tavish9 Thank you for your response. Really appreciate it.

However, I still have doubts. I am unable to still understand how to obtain/render a time variant kplanes-dynamic video. I too have obtained the images but it would be nice to get a video.

I cannot find documentations for the video generation of kplanes-dynamic.

I was able to generate video for static scene using --interpolate only. How else do you suggest I generate a video? Direct ns-render with load-config does not generate any video for me. The other sub commands are {camera-path,interpolate,spiral,dataset} and thats how I decided to go for interpolate choice.

Could you clarify which command you’re referring to?

Additionally, whether using camera-path or interpolate, a video can still be generated with the --output-format video option.

As I mentioned earlier, rendering time-variant kplanes-dynamic requires manual code modifications.

Let me know if this works or if you need further adjustments!

[Anji-Builds]

@Tavish9 Could you explain what manual modification needs to be done for rendering time-variant kplanes-dynamic and how? I am fairly new to this and quite confused.

As per your suggestion I once again tried using ns-render dataset and it renders. However, I am not able to get a video output!

I get a folder lego_dynm.mp4 instead of video, containing another folder test and which contains various image folders.

and all I want is a video and not images of the lego in dynamic motion as per the paper and there official page, with moving lego arm. Where am I going wrong? How do I rectify it. This is the option I see for rendering and choosing dataset is not giving me a video output.

[Tavish9]

Hi, @Anji-Builds, ns-render dataset command does not support video generation. If you want to generate a video, use ns-render interpolate with manual code modification. Here is what you need to do.

1. fix camera times in cameras/camera_paths.py:

   cameras = Cameras(
     ...
     times=times
   )

2. generate times in cameras/camera_utils.py:

   
   time = []
   for idx in range(poses.shape[0] - 1):
     ...
     time += get_interpolated_time(times[id], times[id+1], , steps=steps_per_transition)

return ..., torch.stack(time, dim=0)

3. finish `get_interpolated_time` function.

I plan to submit a pull request to the official repository soon to implement these changes. Stay tuned!