Question about test K-planes with zju-dataset

[louhz]

Hi,Dr. Fridovich-Keil. Thanks for your great work! Right now I am facing some issue when i train and rendering the zju-dataset with K-planes(a human motion dataset). The render result shows below.

After doing some research, I realize that this issue may raised from incorrect transformation matrix coordinate. Since zju-dataset is using opencv coordinate. So I am just curious which camera extrinsic coordinate is used by K-plane: Pytorch3D or OpenGL?

Thanks!

[sarafridov]

We are currently set up to use OpenGL coordinates.

[louhz]

We are currently set up to use OpenGL coordinates.

Thanks!

[Spphire]

嗨，弗里多维奇-凯尔博士。感谢您的出色工作！现在，当我使用 K 平面（人类运动数据集）训练和渲染 zju 数据集时，我遇到了一些问题。渲染结果如下所示。

经过一些研究，我意识到这个问题可能是由不正确的转换矩阵坐标引起的。由于zju数据集使用的是opencv坐标。所以我只是好奇K平面使用哪个相机外在坐标：Pytorch3D还是OpenGL？

谢谢！

does that finally work? can u share me the dataset?

[louhz]

嗨，弗里多维奇-凯尔博士。感谢您的出色工作！现在，当我使用 K 平面（人类运动数据集）训练和渲染 zju 数据集时，我遇到了一些问题。渲染结果如下所示。 经过一些研究，我意识到这个问题可能是由不正确的转换矩阵坐标引起的。由于zju数据集使用的是opencv坐标。所以我只是好奇K平面使用哪个相机外在坐标：Pytorch3D还是OpenGL？ 谢谢！

does that finally work? can u share me the dataset?

Yes, k-plane can deal with the human based dataset with some modification.

The dataset come from the https://github.com/zju3dv/neuralbody#potential-problems-of-provided-smpl-parameters You can reach out authors for the access of the zju-dataset

[OvOtoQAQ]

Hi, did you just replace the videos with zju videos and poses_bounds.npy or did you rewrite dataset? I just assemble a video from zju images and change poses_bound, but it seems something go wrong, please tell me how you did it, thanks a lot.

[louhz]

Hi, did you just replace the videos with zju videos and poses_bounds.npy or did you rewrite dataset? I just assemble a video from zju images and change poses_bound, but it seems something go wrong, please tell me how you did it, thanks a lot.

So, there are two way to implement this, since in the zju-datset, two priors are critical: the smpl and mask. The mask can adapt with k-plane by change the framework and network similar to nerfstudio example. And the smpl prior did not compatible with the k-plane video pipeline since the smpl represent the pose in the 3 dimension, and the k-plane decompose 4D into 6 2D planes with feature. I am still trying to combine the smpl prior with the 4D video. But the static 3D human smpl is compatible with k-plane. What you need to do is to use the smpl to map the human motion in the grid (traditionally in canonical space) and the rest can adapt the k-plane pipeline.

[OvOtoQAQ]

Hi, did you just replace the videos with zju videos and poses_bounds.npy or did you rewrite dataset? I just assemble a video from zju images and change poses_bound, but it seems something go wrong, please tell me how you did it, thanks a lot.

So, there are two way to implement this, since in the zju-datset, two priors are critical: the smpl and mask. The mask can adapt with k-plane by change the framework and network similar to nerfstudio example. And the smpl prior did not compatible with the k-plane video pipeline since the smpl represent the pose in the 3 dimension, and the k-plane decompose 4D into 6 2D planes with feature. I am still trying to combine the smpl prior with the 4D video. But the static 3D human smpl is compatible with k-plane. What you need to do is to use the smpl to map the human motion in the grid (traditionally in canonical space) and the rest can adapt the k-plane pipeline.

Thanks for your detailed explanation! It really helps a lot!