question: how to run on custom data?

[geniuspatrick]

Hi, how to inference on custom data? like a single static scan. Need it be fitted to a spml-x model? If I want to animate this scan, how to get a pose sequence like the "*.npz" file in you demo data? Can you please give the instruction like https://github.com/qianlim/SCALE/issues/6#issuecomment-888125275 ? Again, thanks for your excellent work!

[qianlim]

Hi, in principle the process is the same as that link in SCALE; i.e. first fit a SMPL/SMPLX model to the scan, repose the fitted body model to the desired poses, and generate the the body positional maps of the posed bodies. Then run POP using these positional maps. I'll update this repo in the coming month.

[geniuspatrick]

Hi, I see the great explain in the dataset website. Most fields can be generate by above link in the scale repo, excpet the vtransf. Could you please give a preview code snippet on how to get it? Still, looking forward to the whole pipeline of inference on custom data. Best wishes.

[qianlim]

Hi, sorry for my late reply. Here's a quick how-to in case you need it urgently. To get the vtransf, we modified the smplx package a bit: in lbs.py, this T matrix (shape=[batchsize, number_vertices, 4, 4]) stores the 4x4 transformation matrices from canonical pose space to the posed space for all vertices -- output it (to do so you might need to add a variable for this in the ModelOutput class).

Then, on the UV positional map, each valid pixel corresponds to a point in 3D on the body surface (let's call it 'valid body point' below). The face_id map stores which triangle each valid body point locates. Then use the barycentric coordinates to interpolate the T matrix of the 3 vertices of that triangle -- this interpolated result is the vtransf of the valid body point.

[zhaofang0627]

Hi,

I notice that uv_masks (https://github.com/qianlim/POP/tree/main/assets/uv_masks) only have 3 resolutions, how can I get uv_masks with other resolutions, for example, 512?

Thanks!

[qianlim]

You can use this code (the variable "uv_mask" is what you need) in the SCALE repo to generate the UV map at arbitrary resolution. For that you simply need the template_mesh_<bodymodel>_uv.obj in assets/.

[zhaofang0627]

Thanks! I will try it.

[caiyongqi]

Hi, sorry for my late reply. Here's a quick how-to in case you need it urgently. To get the vtransf, we modified the smplx package a bit: in lbs.py, this T matrix (shape=[batchsize, number_vertices, 4, 4]) stores the 4x4 transformation matrices from canonical pose space to the posed space for all vertices -- output it (to do so you might need to add a variable for this in the ModelOutput class).

Then, on the UV positional map, each valid pixel corresponds to a point in 3D on the body surface (let's call it 'valid body point' below). The face_id map stores which triangle each valid body point locates. Then use the barycentric coordinates to interpolate the T matrix of the 3 vertices of that triangle -- this interpolated result is the vtransf of the valid body point.

Hi, So we only use the rotation part of T, not the translation? Thank you! In dataset.py: if vtransf.shape[-1] == 4: vtransf = vtransf[:, :3, :3]

[qianlim]

Right

[caiyongqi]

Hi, I have a new issue. Is the vtranf calculated from the input position map (without SMPL shape parameters) or from the query position map (with SMPL shape parameter)? They seem to have slight differences.