how to load the deepfashion3d data?

[merlinyx]

Hi! I have a very basic question on loading the data to view. I was trying to view the point cloud, mesh, and the body (from the v2 release) using meshplot but ran into the following issues:

1. not sure which SMPL pkl to use, I don't know where to find the info for whether the body is female, male, or neutral?
2. the packed_pose has field "pose", "scale", and "trans"; I used "pose" and "trans" to initialize the smpl model and then scaled it with the "scale" float but the resulting body mesh and garment mesh do not line up when I load them into the same meshplot plot:

The code I used to create this plot is as follows:

import smplx
import torch
import numpy as np
import pickle
import meshplot as mp
import open3d as o3d
import igl

df3d_folder = "../../../Downloads/data/deepfashion3dv2"
pose_folder = os.path.join(df3d_folder, "packed_pose")
pc_folder = os.path.join(df3d_folder, "point_cloud")
fl_folder = os.path.join(df3d_folder, "featureline_annotation")
mesh_folder = os.path.join(df3d_folder, "filtered_registered_mesh")

garment_id = 19 # the first in the short-sleeve upper category
garment_id = str(garment_id)

available_poses = [filename.split(".")[0] for filename in os.listdir(os.path.join(pose_folder, garment_id))]
available_poses = sorted(available_poses)
# available_poses

pose_id = available_poses[0]
fl_id = "_".join(pose_id.split("-"))
# pose_id, fl_id

# load pose and see if I can reconstruct the body
with open(os.path.join(pose_folder, garment_id, pose_id + ".pkl"), "rb") as f:
    packed_pose = pickle.load(f)
# packed_pose

model_n = smplx.create("./models/", model_type="smpl", gender="female",
                       global_orient=packed_pose["pose"][:3].reshape(1, 3),
                       body_pose=packed_pose["pose"][3:].reshape(1, 69),
                       transl=packed_pose["trans"].reshape(1, 3),
                       dtype=torch.float64)

output_n = model_n()
verts_n = output_n.vertices.detach().cpu().numpy().squeeze()
faces_n = model_n.faces

# newvn = (verts_n - offset) * packed_pose["scale"]# + offset
newvn = verts_n * packed_pose["scale"]

# next try to load the .ply files
pcd = o3d.io.read_point_cloud(os.path.join(pc_folder, garment_id, pose_id + ".ply"))

# visualize the garment mesh on body
v, f = igl.read_triangle_mesh(os.path.join(mesh_folder, pose_id, "model_cleaned.obj"))

p = mp.plot(verts_n, faces_n, c=np.array([0.8,1,0.8]))
p.add_mesh(v, f)
p.add_points(np.asarray(pcd.points))

If anyone could help me understand how to correctly load the data to view them that'll be great! Thanks!

[kv2000]

Hi, sorry for the delayed reply:

1. We use the ‘male’ model to initialize the SMPL body.
2. The underlying SMPL body mesh could be computed through: SMPL (pose, shape) * scale + trans.

The resulting SMPL body would roughly align with the garment point cloud. However, it could also happen that the garment point cloud is slightly larger/smaller than the underlying SMPL body.

[merlinyx]

I see, thank you for the reply!

[msverma101]

hi what did you use for shape here.. the dataset only has pose no shape in the packed_pose pkl files

[merlinyx]

that's true! I think if no shape param is given to smplx create() it'll just use some default shape parameters.

[msverma101]

So i have two main problems.

1. The default shape is little bigger than the garment.
2. There is some misalignment, the translation/global orient might have some errors see attached photo

[merlinyx]

Ah yes I think according to their response it's possible that it's slightly misaligned...

[msverma101]

its quiet a bit off. how am i to use this to do some evaluation if i am stuck finding out the exact place where the pose should to fit the cloth. could you give me a better solution such that if i use the pose and the garment and set it they should seamlessly set on top of each other