Got bad rendering quality than expected after training 4K4D on NHR dataset

[YijieYang23]

Hi, I followed the readme tutorial, prepared the full NHR dataset and run the training scripts:

evc-train -c configs/exps/4k4d/4k4d_basketball_r4.yaml

after 1600 training epoch was done, I found bad rendering quality in eval results, like this: the computed metrics are as follows:

{
    "summary": {
        "psnr_mean": 30.552193959554035,
        "psnr_std": 0.9542468203310044,
        "ssim_mean": 0.9624246756235758,
        "ssim_std": 0.005160357162961342,
        "lpips_mean": 0.10635496800144513,
        "lpips_std": 0.006034270129439567
    },
    "metrics": [
        {
            "psnr": 31.88111114501953,
            "ssim": 0.9669070243835449,
            "lpips": 0.10305734723806381,
            "camera": 0,
            "frame": 0
        },
        {
            "psnr": 31.195079803466797,
            "ssim": 0.9596953392028809,
            "lpips": 0.10957354307174683,
            "camera": 0,
            "frame": 100
        },
        {
            "psnr": 31.23896026611328,
            "ssim": 0.9714922904968262,
            "lpips": 0.09953031688928604,
            "camera": 26,
            "frame": 0
        },
        {
            "psnr": 29.63504981994629,
            "ssim": 0.9612285494804382,
            "lpips": 0.11462226510047913,
            "camera": 26,
            "frame": 100
        },
        {
            "psnr": 29.204450607299805,
            "ssim": 0.9586771130561829,
            "lpips": 0.099294014275074,
            "camera": 52,
            "frame": 0
        },
        {
            "psnr": 30.158512115478516,
            "ssim": 0.956547737121582,
            "lpips": 0.112052321434021,
            "camera": 52,
            "frame": 100
        }
    ]
}

I generate the realtime version model:

python scripts/realtime4dv/charger.py --sampler SuperChargedR4DV --exp_name 4k4d_basketball -- -c configs/exps/4k4d/4k4d_basketball_r4.yaml,configs/specs/super.yaml

and use the converted model to render novel views:

evc-test -c configs/projects/realtime4dv/rendering/4k4d_basketball.yaml,configs/specs/eval.yaml,configs/specs/spiral.yaml,configs/specs/ibr.yaml

The rendering video is as follows:

https://github.com/user-attachments/assets/9f93de80-f0b8-4f14-9abd-ea7ddd9907ec

It appears that the points on the human body are quite big and noisy. Additionally, there are many outlier small points scattering outside the region of the human body.

Could you give some suggestion on what might be causing the bad result?

[YijieYang23]

Following the same procedure, I trained the DNA-Rendering: 4k4d_0013_01 sample and rendered novel views. The rendering results turned out to be very good and met expectations.

https://github.com/user-attachments/assets/92361b35-d86b-49ae-83c5-bbc1d026a189

[dendenxu]

Hi, thanks for using our code! The basketball results indeed look very strange. I tried reproducing but the training looks normal on my end. Could you please try checking the training recording using tensorboard (the log should be located at data/record/4k4d_basketball_r4) and check whether there's some instability? In some rare occasions, the loss might suddenly soar and never go down. If that's the case, a possible solution is to use the more stable CUDA-based rasterizer (enable using render_gs=True, the full command line should be:).

evc-train -c configs/exps/4k4d/4k4d_basketball_r4.yaml model_cfg.sampler_cfg.render_gs=True

[YijieYang23]

Yes! I checked the training log using TensorBoard. In contrast to other normal experiments (e.g., 4k4d_0013_01), NHR basketball experiment shows a significant fluctuation and upward trending loss, especially for the msk_loss. When I enabled render_gs=True, this fluctuation and upward trend in the loss did not occur, and I obtained good rendering results. Thank you very much for resolving the problem.

Additionally, I would like to inquire if you have any plans to further complete the Optimizing the Cameras section in the README. Would it be possible for you to provide some concise guidance here?

[dendenxu]

Good to hear the issue is resolved! The TLDR is to export the optimized camera using this script: https://github.com/zju3dv/4K4D/blob/main/scripts/tools/extract_optimized_cameras.py and use the config entry dataloader_cfg.dataset_cfg.intri_file and extri_file to select the optimized cameras, and do the same for val_dataloader. We've updated the way of loading and using the optimized camera since release and forgot to update the readme. We'll update this shortly! Thanks for the heads-up!

[brianneoberson]

Hi, I had the same problem on NHR basketball and tried training with model_cfg.sampler_cfg.render_gs=True but the results were not better at all. The renders ended up being completely transparent. Does someone have an idea why this is happening?