Standard train and test set split

[boycehbz]

Thanks for the great work! Do you have a standard split for the train and test sets?

[MoyGcc]

Hi, thanks for your interest. We position our Hi4D dataset as a test benchmark thus we didn't train or fine-tune SOTA SMPL fitting methods on our dataset.

Due to our positioning of Hi4D, we don't provide explicit splits of the dataset into train/val/test. You could split the dataset so that it fits your setting well. There is no specific tip from our side for the splits as long as the splits are shuffled randomly and a ratio such as 8:2 is met.

[boycehbz]

Thanks so much for your quick reply! I have another question about the annotations. I project the SMPL mesh to the image plane. Some parts do not seem to align with the image observations. Is it correct?

[MoyGcc]

I doubt this is because you might undistort the image again and then project the SMPL onto it. The provided images should be undistorted already.

If this doesn't resolve your question, can you please share your visualization code and the sequence/frame of the image you showed?

[boycehbz]

I directly use the annotations and images provided in the downloaded dataset. The following is the visualization code ` import sys sys.path.append('./') from utils.FileLoaders import * from utils.smpl_torch_batch import SMPLModel import torch import cv2

def surface_projection(vertices, faces, joint, extri, intri, image, viz=False):

  im = image
  h = im.shape[0]
  # homogeneous
  intri_ = np.insert(intri,3,values=0.,axis=1)
  temp_v = np.insert(vertices,3,values=1.,axis=1).transpose((1,0))

  # projection
  out_point = np.dot(extri, temp_v)
  dis = out_point[2]
  out_point = (np.dot(intri_, out_point) / dis)[:-1]
  out_point = (out_point.astype(np.int32)).transpose(1,0)

  # color
  max = dis.max()
  min = dis.min()
  t = 255./(max-min)
  color = (255, 255, 255)

  # draw mesh
  for f in faces:
      point = out_point[f]
      im = cv2.polylines(im, [point], True, color, 2)

  # joints projection
  temp_joint = np.insert(joint,3,values=1.,axis=1).transpose((1,0))
  out_point = np.dot(extri, temp_joint)
  dis = out_point[2]
  out_point = (np.dot(intri_, out_point) / dis)[:-1].astype(np.int32)
  out_point = out_point.transpose(1,0)

  # visualization
  if viz:
      ratiox = 800/int(im.shape[0])
      ratioy = 800/int(im.shape[1])
      if ratiox < ratioy:
          ratio = ratiox
      else:
          ratio = ratioy

      cv2.namedWindow("mesh",0)
      cv2.resizeWindow("mesh",int(im.shape[1]*ratio),int(im.shape[0]*ratio))
      cv2.moveWindow("mesh",0,0)
      cv2.imshow('mesh',im/255.)
      cv2.waitKey()

  return out_point, im

smpl = SMPLModel()

camera = load_npz('/media/buzhenhuang/HDD/Human-data-source/Hi4D/_pair00_1/pair00/dance00/cameras/rgb_cameras.npz') intris = camera['intrinsics.npy'] extris = camera['extrinsics.npy'] dists = camera['dist_coeffs.npy']

intri = intris[0] extri = np.eye(4) extri[:3] = extris[0]

params = load_npz('/media/buzhenhuang/HDD/Human-data-source/Hi4D/_pair00_1/pair00/dance00/smpl/000010.npz')

img = cv2.imread('/media/buzhenhuang/HDD/Human-data-source/Hi4D/_pair00_1/pair00/dance00/images/4/000010.jpg')

betas = params['betas.npy'] global_orient = params['global_orient.npy'] body_pose = params['body_pose.npy'] transl = params['transl.npy'] pose = np.concatenate((global_orient, body_pose), axis=1)

pose = torch.from_numpy(pose).reshape(-1, 72).float() betas = torch.from_numpy(betas).reshape(-1, 10).float() transl = torch.from_numpy(transl).reshape(-1, 3).float()

verts, joints = smpl(betas, pose, transl)

surface_projection(verts[1].detach().numpy(), smpl.faces, joints[1].detach().numpy(), extri, intri, img, viz=True)

`

[MoyGcc]

Thanks for the information. I just quickly tried from my side and it seems that there is no problem aligning the SMPL with the images. I provide a very easy code segment for you to try out. Hope that could help you during debugging.

import cv2
import numpy as np
from smplx import SMPL

img = cv2.imread(f'/home/chen/disk2/Hi4D/pair00/dance00/images/4/000010.jpg')

smpl = SMPL('/home/chen/Models/smpl', gender='female')
smpl_file = dict(np.load('/home/chen/disk2/Hi4D/pair00/dance00/smpl/000010.npz'))
cam = np.load('/home/chen/disk2/Hi4D/pair00/dance00/cameras/rgb_cameras.npz')
# cam 4 is the 0-th camera
cam_0_intrinsics = cam['intrinsics'][0]
cam_0_extrinsics = cam['extrinsics'][0]

smpl_verts = smpl_file['verts'][1]
P = cam_0_intrinsics @ cam_0_extrinsics

for j in range(0, smpl_verts.shape[0]):
    padded_v = np.pad(smpl_verts[j], (0,1), 'constant', constant_values=(0,1))
    temp = P @ padded_v.T
    pix = (temp/temp[2])[:2]
    output_img = cv2.circle(img, tuple(pix.astype(np.int32)), 3, (0,255,255), -1)

cv2.imwrite('./hi4d_proj.png', output_img)

[boycehbz]

Thanks! I used a neutral SMPL model and got the wrong results. The problem is solved by replacing the model with a male one. But why is the gender female in your code?

[MoyGcc]

I was initially testing another sequence where the subjects were all female and forgot to change it back to male for this sequence. There isn't a big difference between the gender-specific SMPL models but the neutral model is quite different from both male and female models. Glad it has been resolved. Closing this issue.