H36m training requirements -

[Dipankar1997161]

Hello @taoranyi,

I appreciate your work here. I also planning to train humannerf on h36m dataset, atleast for 1 subject. Could you please tell me, how can we get the h36m smpl parameters and the necessary mask

I tried to use the Easymocap file, since h36m provides the 3d groundtruth https://github.com/zju3dv/EasyMocap/blob/master/apps/demo/smpl_from_keypoints.py (Is this the correct way?)

So I got the smpl parameters like this as an example: I used the h36m Positions_3D_mono files (are these correct files)

  {  "id": 1611,
    "Rh": [
      [-1.109, 4.132, -2.318]
    ],
    "Th": [
      [-246.794, 285.948, 657.016]
    ],
    "poses": [
      [0.000, 0.000, 0.000, 0.917, -1.202, 1.189, 0.836, 1.005, 0.382, 2.144, -0.020, -0.395, -0.627, -1.443, 0.152, ...]
    ],
    "shapes": [
      [11905.026, -2553.326, 698.123, -111.642, 616.076, 136.446, -253.905, -107.915, 117.896, 356.510]
    ]
}

then I generated the mask using SAM -

I used the prepare_dataset.py for the humannerf wild/monocular since I am using a single camera from h36m. after all that, I got the data loader error, when I placed num_workers to 0,

The near, far list are empty which are generated from the "joints". The bbox values are vast: min x: -247 max x: +256

So, I need little help on what I could be doing wrong here, is it the masking issue or smpl issue or something? Really need your help, since you already trained on h36m.

Thank you once again. I hope you can help me out

[taoranyi]

Hello, thank you for your attention. 1、I have not used smpl_from_keypoints.py before, so I am not sure about its accuracy. If you want to obtain the smpl parameters defined by easymocap from the standard smpl parameters, you can refer to this script, which is definitely correct. Or you can directly obtain the data from here, which has been processed by animatable_nerf and is in the same format as zju_mocap. 2、For the smpl parameters defined by easymocap, you should use the script inside the humannerf prepare_zju_mocap. The prepare_wild script is for the standard smpl.

[Dipankar1997161]

Hello, thank you for your attention. 1、I have not used smpl_from_keypoints.py before, so I am not sure about its accuracy. If you want to obtain the smpl parameters defined by easymocap from the standard smpl parameters, you can refer to this script, which is definitely correct. Or you can directly obtain the data from here, which has been processed by animatable_nerf and is in the same format as zju_mocap. 2、For the smpl parameters defined by easymocap, you should use the script inside the humannerf prepare_zju_mocap. The prepare_wild script is for the standard smpl.

Thank you for the quick response. I have emailed him for thr process data of h36m. Could you tell me, where did you obtain the masks for rendering, since humanerf requires masks as well as images.

The masks should be alpha masks right as shown at the top or should it be the background removal ones.

If you can provide this info, I can work further. Thank you once again

[taoranyi]

Hello, the masks generated by you looks fine. Most of the techniques for human segmentation can be used without any special requirements. The original H36m dataset provides the correct masks, and the dataset processed in animatable_nerf also includes the masks.

[Dipankar1997161]

Hello, the masks generated by you looks fine. Most of the techniques for human segmentation can be used without any special requirements. The original H36m dataset provides the correct masks, and the dataset processed in animatable_nerf also includes the masks.

😃 Thank you so much. I just received the files from Pengsida. Now I believe, I have to run the script in prepare_zju-mocap, since their smpl is from Easymocap version.

Should I change the preprocessing script for H36m, since the cameras are different.

Can I use the processing script you providing or are there any special requirements. It looks the same to me Do let me know if any changes are needed

Thank you once again for your help

[taoranyi]

Hello, just make sure to include the view ID in the naming to avoid overwriting due to identical names. There is no difference in other aspects.

[Dipankar1997161]

@taoranyi , Hello,

In the prepare_zjumocap script, there are 2 mask files needed, "masks" and "masks_cihp" - can you tell me the difference. As per what I saw, masks_cihp is more of a blend and masks is just black image. If there's more, let me know.

What I want is to generate the "Masks" as per the zju-mocap dataset, so the prepare_dataset can run since it requires that file. How Can I attain that mask? Is it just a Black Image of the same size as the original image?

Because the processed file I received has "cache_masks" and "masks_cihp", both look identical to me. Then there's a file (prepare_mask.py) not sure If i should run this.

Mask_cihp

Cache_Mask

Do let me know, Thank you

[Dipankar1997161]

@taoranyi

I noticed another thing,

I believe it is necessary to have smpl parameters for every frame. The processed data takes every 5th frame smpl parameters. Won't this create any imbalance for training?

If you have any processed file could you share or any solutions for the above mentioned this

[Dipankar1997161]

@taoranyi , Hello,

In the prepare_zjumocap script, there are 2 mask files needed, "masks" and "masks_cihp" - can you tell me the difference. As per what I saw, masks_cihp is more of a blend and masks is just black image. If there's more, let me know.

What I want is to generate the "Masks" as per the zju-mocap dataset, so the prepare_dataset can run since it requires that file. How Can I attain that mask? Is it just a Black Image of the same size as the original image?

Because the processed file I received has "cache_masks" and "masks_cihp", both look identical to me. Then there's a file (prepare_mask.py) not sure If i should run this.

Mask_cihp

Cache_Mask

Do let me know, Thank you

I resolved the Masking problem, which I had. The training actually started.

Just share your thoughts on the this problem: https://github.com/hustvl/GNeuVox/issues/4#issuecomment-1577058560

[taoranyi]

Hello, I haven't encountered the issue of unbalanced training, and I guess it's because his movement is slow.

[Dipankar1997161]

@taoranyi,

Hello, I tried to process on H36m video from ROMP(I am using the processed h36m file from ROMP). But when I trained the file, it was really bad.

At first, the images were clipped in half but setting the camera extrinsic to np.eye(4) rendered the images in full.

However, I am unsure of the camera intrinsic values to use, so i used fx,fy = 443.4(given in ROMP config file with FOV = 60) and cx, cy = 512

The mask is correct, but something might be the issue (I believe its the camera values which is causing the issue here.)

Could you check this once. I think we cannot use ROMP for h36m since it has 4 sync cameras. I used the animetabl nerf processed file, but i want to further generate such file for another action.

Would be great if we can discuss on this a bit