So i have two main problems. Issue alligning the garment on top of the smpl pose mentioned

[msverma101]

          So i have two main problems.

1. The default shape is little bigger than the garment.
2. There is some misalignment, the translation/global orient might have some errors see attached photo

Originally posted by @msverma101 in https://github.com/GAP-LAB-CUHK-SZ/deepFashion3D/issues/11#issuecomment-1710823035

[kv2000]

Hi, thanks for your interest in the dataset :D.

1. As for the shape parameters, we are not optimizing the SMPL shape parameters (so the shape parameter for the SMPL models in the dataset is always set to one) ...
   It would happen that the topology consistent garment is slightly larger/smaller than the underlying SMPL, but it should be roughly aligned cuz we've tried to manually clean it up/aligned.

2. As for the alignment, would you please provide the garment ID that you showed? Moreover, there is a scaling factor in the provided SMPL parameter file. And the resulting SMPL body mesh could be computed through -> scale * SMPL(pose, shape(set to zero) + translation

[msverma101]

this is 1-1 in the first one and this is the code below i also tried it for 2-1 as well just to confirm if there was a problem with the first one

import smplx
import torch
import numpy as np
import pickle

import open3d as o3d
import igl
import os

df3d_folder = "/home/dataset/fashion3dv2"
pose_folder = os.path.join(df3d_folder, "packed_pose")
pc_folder = os.path.join(df3d_folder, "point_cloud")
fl_folder = os.path.join(df3d_folder, "featureline_annotation")
mesh_folder = os.path.join(df3d_folder, "filtered_registered_mesh")

garment_id = 2 # the first in the short-sleeve upper category
garment_id = str(garment_id)

available_poses = [filename.split(".")[0] for filename in os.listdir(os.path.join(pose_folder, garment_id))]
available_poses = sorted(available_poses)
# available_poses

pose_id = available_poses[0]
print(pose_id)
fl_id = "_".join(pose_id.split("-"))
# pose_id, fl_id

# load pose and see if I can reconstruct the body
with open(os.path.join(pose_folder, garment_id, pose_id + ".pkl"), "rb") as f:
    packed_pose = pickle.load(f)
# packed_pose

to_tensor = lambda d : {key: torch.Tensor(val) for key, val in d.items() if key not in ["scale"]}
# packed_pose = to_tensor(packed_pose)

model_n = smplx.create("/home/ICON/data/smpl_related/models/", model_type="smpl", gender="male")

output_n = model_n(global_orient=torch.from_numpy(packed_pose["pose"][:3].reshape(1, -1, 3)), 
                   body_pose=torch.from_numpy(packed_pose["pose"][3:].reshape(1,-1, 3)), 

                       dtype=torch.float64,)

output_n.vertices = output_n.vertices * packed_pose["scale"] + torch.from_numpy(packed_pose["trans"].reshape(1, 3))

import trimesh
verts_n = output_n.vertices.detach().cpu().numpy().squeeze()
faces_n = model_n.faces
mesh = trimesh.Trimesh(vertices= verts_n, faces= faces_n)
mesh.export("smpl.obj")

[msverma101]

for clarification, I am smplx library for doing skinning

[kv2000]

Hi, i've tested on my side with SMPL (I haven't tried with SMPLX), more specifically, SMPL_Layer from smplpytorch.pytorch.smpl_layer, and it seems to work ( the following is the result with 1-1, and roughly aligns)

       smpl_layer = SMPL_Layer(
            center_idx=0,
            gender='male',
            model_root='smpl_models'
        )

        src_dict = pkl.load(open(os.path.join(src_dir,each_cloth,each_cloth_pose),'rb'))        
        pose = src_dict['pose']
        trans = src_dict['trans']
        scale = src_dict['scale']

        fin_pose= torch.FloatTensor(pose).unsqueeze(0)
        fin_pose = fin_pose.cuda()

        fin_shape = torch.zeros((1,10)).float()
        fin_shape = fin_shape.cuda()

        ret_verts, ret_jtr = smpl_layer(fin_pose, fin_shape)

        ret_verts = ret_verts.detach().cpu().numpy()[0]

        trans_verts = ret_verts * scale + trans

[msverma101]

could you please give me the code in accordance with the folder format of deepfashion3dv2 and also where can i get fin_pose_mask tensor?

[kv2000]

ffor the following line:

src_dict = pkl.load(open(os.path.join(src_dir,each_cloth,each_cloth_pose),'rb'))

just switch the file name to any of the pose files (.pkl), then you're good to go.. no fin_pose_mask is needed (i've edited the original comment)

[msverma101]

yes it worked now. better than expected

[kv2000]

great :D

[msverma101]

here is the code for it

from smplpytorch.pytorch.smpl_layer import SMPL_Layer
# from display_utils import display_model
import pickle as pkl
import os
import torch

df3d_folder = "/home/dataset/fashion3dv2/fashion3dv2"
pose_folder = os.path.join(df3d_folder, "packed_pose")
pc_folder = os.path.join(df3d_folder, "point_cloud")
fl_folder = os.path.join(df3d_folder, "featureline_annotation")
mesh_folder = os.path.join(df3d_folder, "filtered_registered_mesh")

garment_id = 63 # the first in the short-sleeve upper category
garment_id = str(garment_id)

available_poses = [filename.split(".")[0] for filename in os.listdir(os.path.join(pose_folder, garment_id))]
available_poses = sorted(available_poses)
# available_poses

pose_id = available_poses[0]
fl_id = "_".join(pose_id.split("-"))

# load pose and see if I can reconstruct the body
with open(os.path.join(pose_folder, garment_id, pose_id + ".pkl"), "rb") as f:
    src_dict = pkl.load(f)

smpl_layer = SMPL_Layer(
    center_idx=0,
    gender='male',
    model_root='/home/dataset/fashion3dv2/smpl/',

)

pose = src_dict['pose']
trans = src_dict['trans']
scale = src_dict['scale']

fin_pose= torch.FloatTensor(pose).unsqueeze(0)
# fin_pose = fin_pose_mask.cuda()

fin_shape = torch.zeros((1,10)).float()
fin_shape = fin_shape.cuda()

ret_verts, ret_jtr = smpl_layer(fin_pose, fin_shape)

ret_verts = ret_verts.detach().cpu().numpy()[0]

trans_verts = ret_verts * scale + trans

import trimesh
mesh = trimesh.Trimesh(vertices= trans_verts, faces= smpl_layer.th_faces.detach().cpu().numpy())
mesh.export("smpl1.obj")

[msverma101]

hi i would say there is still a problem i want to use them as a quantitative results to benchmark some models and the smpl is projecting outside the cloth and most of the clothes are actually for women but only the men smpl fits a in terms of height but the garment in colliding with it. is there a better way such that the smpl doesnt cooincide? if not the dataset wont be of any help to be