Blurry rendered images in the viewer

[dlazares]

Hi I got the viewer up and running. The video starts very clear off the initial ply but seems to get blurry and fall apart as time progresses forwards. In the paper, you report seeing a good average PSNR across the sequence so I'm wondering if something is wrong on the viewer side? here's an example on one of the later frames in the video

[SJoJoK]

Hello @dlazares, I capture a video by my phone (I'm not familiar with the OBS Studio, so the quality of the screen recording is low) to show how to use the viewer, please check the attached video. Note that the directories called per-frame_3dgs and raw are not used by the viewer, as you can see in the code. Please let me know if you follow step shows in the videos but still get buggy renderings, it may due to I uploaded the wrong files in the Google Drive. Thanks.

https://github.com/SJoJoK/3DGStream/assets/50450335/033493a4-8967-49a4-be00-377a2d02b4c9

[dlazares]

It may be wrong files in Google drive since I did the steps you did

[SJoJoK]

Hello @dlazares , I've just downloaded the FVV that I previously uploaded to Google Drive. It works well with the viewer, so I think the problem might be due to the mismatch of the environment or something else.

Here's some information of my machine, FYI:

• OS: Windows 11
• GPU: RTX 2060
• Driver: 531.87
• CUDA: 12.1
• Python: 3.10
• Pytorch: 2.1.1+cu121
• tinycudann: 1.7

Besides, I test the viewer and the FVV in another machine (Actually, the FVV is constructed/trained on this machine), and it also works well. Here's the information of it:

• OS: Ubuntu 22.04
• GPU: RTX A6000
• Driver: 535.86.05
• CUDA: 11.8
• Python: 3.8
• Pytorch: 2.0.1+cu118
• tinycudann: 1.7

[dlazares]

hmmm.... my setup is similar to your first one except I'm running on Windows Subystem for Linux and am using a 4090.

GPU is diffent and driver version (551.86) but I don't think that's the cause since it renders out

[SJoJoK]

The driver's version might be the reason since the NTC is implemented by tinycudann, a GPU/Driver related module. The wrong output of NTC will make the scene deform incorrectly. As the error of deformation accumulates, the latter frames will be buggy.

Since the implementation of the viewer is quite simple and I've tested it on different machines, I think rendering process of the viewer is bug-free. Thus, the reason of your buggy rendering can only due to the wrong output of the NTC (In your image, the additional 3dgs is rendered correctly). Upon the release of our training code, you can construct your own FVV (init_3dgs, NTCs and additional_3dgs) and render it by the viewer, as the correctness of the NTC have been validated in your training process.

[dlazares]

Sounds good, I'll give it a try when the training code is up.