Processing custom datasets

[vipinsdk]

Hello Shenhan,

Thanks for sharing the preprocessing code for Gaussian Avatars paper. I want to train the Gaussian Avatars model with my own custom dataset. I am capturing a multi-view dataset consisting of 6 cameras and the I have done the camera calibration and have the camera parameters in OPENCV "yml" file format consisting all the parameters from 6 cameras in two files, one for intrinsics and other one for extrinsics. I want to use this recorded videos and preprocess it using this repository. I want to know how I should structure my data to run this pipeline. I saw the format described for nerRSemble but I still have few questions:

1. What is the exact input to this pipeline, is it just videos and their corresponding camera parameters? What else do I need to provide as input for this pipeline?
2. Do I need to also provide background images for each of the camera?
3. This code will generate the facial landmarks based on the dataset right?

Finally, I already have access to NeRSermble dataset from Gaussian Avatars paper, does the dataset here differ from that paper.

Thanks and Regards Vippin

[ShenhanQian]

Hi Vippin, thanks for your questions.

1. Basically, videos and camera parameters are all you need. NeRSemble dataset also provides color correction data for each view. But you can ignore it if you don't have this data.
2. Preferrably, providing background images for each camera allows you to use BackGroundMattingV2. It produces better matting results than RobustVideoMatting, which does not require background images. This only relates to preprocessing of the data.
3. Yes, the landmarks will be generated on the first run and stored with the preprocessed data for re-use.

Finally, videos in NeRSemble undergo a mild compression before release. The difference can be ignored.

[vipinsdk]

Hello Shenhan,

Thanks for the reply. I have managed to do get my custom dataset to the required format and now I want to train the model but I am facing issues with NVDiffRast

[F919 03:43:17.854078983 glutil.cpp:338] eglInitialize() failed
Aborted (core dumped)

I am trying to run the code on a cluster and it does not have display. I saw some fixes online but none of them work and eventually I get this error.

For now: I thought of a workaround of disabling the use_opengl flag to False but I am not sure if it will make any difference or if it is even recommended to do so.

Please let me know if there is a fix for this issue.

Thanks and Regards, Vippin

[ShenhanQian]

Setting use_opengl to false will switch nvdiffrast to its CUDA backend. It works similarly but has a stricter limitation on the resolution:

The CUDA rasterizer does not support output resolutions greater than 2048×2048, and both dimensions must be multiples of 8.

Given that the OpenGL backend is more flexible, I would still recommend to fix the environment. From my personal experiences of deploying this repo on different clusters, it should usually work if you can let glxgears run smoothly. For a machine without monitor, you can still do this with X11 forwarding.

[vipinsdk]

Hello Shenhan,

Thanks for the reply. I was successful in making the code run in server without an display by setting export EGL_PLATFORM="surfaceless" but the code crashes with the following error:

Traceback (most recent call last):
  File "/home/jeetmal/VHAP/vhap/track_nersemble.py", line 21, in <module>
    tracker.optimize()
  File "/home/jeetmal/VHAP/vhap/model/tracker.py", line 1479, in optimize
    self.optimize_stage('rgb_init_texture', sample)
  File "/home/jeetmal/VHAP/vhap/model/tracker.py", line 1515, in optimize_stage
    self.optimize_iter(sample, optimizer, stage)
  File "/home/jeetmal/VHAP/vhap/model/tracker.py", line 1543, in optimize_iter
    ) = self.compute_energy(
  File "/home/jeetmal/VHAP/vhap/model/tracker.py", line 825, in compute_energy
    rast_dict = self.rasterize_flame(
  File "/home/jeetmal/VHAP/vhap/model/tracker.py", line 361, in rasterize_flame
    rast_dict = self.render.rasterize(verts, faces, RT, K, image_size, False, train_mode)
  File "/home/jeetmal/VHAP/vhap/util/render_nvdiffrast.py", line 224, in rasterize
    rast_out, rast_out_db = self.rasterize_fragments(verts_clip, tri, image_size, use_cache, require_grad)
  File "/home/jeetmal/VHAP/vhap/util/render_nvdiffrast.py", line 247, in rasterize_fragments
    rast_out, rast_out_db = dr.rasterize(self.glctx, verts_clip, tri, image_size)
  File "/root/miniconda3/envs/VHAP/lib/python3.10/site-packages/nvdiffrast/torch/ops.py", line 310, in rasterize
    return _rasterize_func.apply(glctx, pos, tri, resolution, ranges, grad_db, -1)
  File "/root/miniconda3/envs/VHAP/lib/python3.10/site-packages/torch/autograd/function.py", line 574, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/root/miniconda3/envs/VHAP/lib/python3.10/site-packages/nvdiffrast/torch/ops.py", line 246, in forward
    out, out_db = _get_plugin(gl=True).rasterize_fwd_gl(raster_ctx.cpp_wrapper, pos, tri, resolution, ranges, peeling_idx)
RuntimeError: Cuda error: 304[cudaGraphicsGLRegisterBuffer(&s.cudaPosBuffer, s.glPosBuffer, cudaGraphicsRegisterFlagsWriteDiscard);]

I am not sure, why this error occurs. I know that this NVDiffRast error but I have not changed the code so I assume it would run without errors. Have you also faced this error and can you let me know how this can be solved.

I am using the same package version as mentioned in the README.md Following are the version of CUDA: nvidia-smi - 12.6 nvcc - 12.1 (inside the conda env)

Thanks in advance

Regards, Vippin