Human reconstruction from point cloud captured with kinect Azure

[kfarivar]

Hi,

Since there was no script for human reconstruction, I wrote a script and followed the pipeline in the readme file. These are the results I get which are in no way as good as the examples you have provided in the paper.

Here is my script: (I put it in the main directory of ifnet and ran it from there, tha paths need to be corrected)

kiya_recounstruct_pc.txt

here is the point cloud I used (it's a CSV file):

frame_20.txt

I should mention that for the reconstruction to work the point cloud should have a specific orientation. (since Conv layers are not rotation invariant). The head should be in the direction of positive y, and the human should face the positive z (I guess). (so I have done this for my point cloud)

This is my result with default parameters (points sampled from the point cloud 3,000 input res 125, output res: 256)

This is the best result I got, which is just a little better: ( points sampled from the point cloud 20,000 input res 256, output res: 400)

I was wondering if I'm doing something wrong ? and what hyperparameters you have used (res, sample size, retrieval_res etc) to reconstruct the examples in the paper (say the BUFF dataset) ?

Maybe it is the case that the point clouds the network was trained on and the ones from kinect are different, since if I remember correctly you synthetically create your point clouds from the scans. But the difference between the qualities of reconstruction is a lot and I don't think this could solely explain the problem.

I was also wondering if you were also using the same checkpoint you have provided here (for the SVR model) to do your reconstructions ?

thanks

[Bingoang]

Wow, you 're so cool! Thanks for you script for human reconstruction! And can you tell us the command and related params of running the script?

[kfarivar]

So I have tried to reconstruct a human that was facing the camera from the side and the results are even worse. I guess these cases were not in the training set ?

here is the point cloud file: frame_340.txt

input:

output:

[kfarivar]

Wow, you 're so cool! Thanks for you script for human reconstruction! And can you tell us the command and related params of running the script?

there are no params. you can look at my code (it is completely commented) and you need to change the paths according to where your files are. also change the parameters to what you want. I put the script in the main directory of ifnet and executed it from there.

[Bingoang]

Wow, you 're so cool! Thanks for you script for human reconstruction! And can you tell us the command and related params of running the script?

there are no params. you can look at my code (it is completely commented) and you need to change the paths according to where your files are. also change the parameters to what you want. I put the script in the main directory of ifnet and executed it from there.

Thanks very much for your quick and detailed reply! It really works! Your script is so nice to us fans of if-net! Exciting~:-D I'm also expecting Julian's reply to your issue~

[jchibane]

Hi all,

thanks for reaching out!

The IF-Net models I released for SVR were trained to retain the details of the input data and complete the missing parts.

The completed back side will have realistic human shape but will be smooth. The front should have all the details that the input data had. Here an example for BUFF: Input

Output

As you mentioned, rotation (but also scaling) does matter. The rotation you chose is right. For generating the input to the network for BUFF, I attached you the preprocessing script. It goes from a full mesh, to a partial 3D point cloud (similar to a your Kinect data - but more detailed, sharper point clouds). And it put's it in the format for the network - I also added the script, how to read such data in. ( Also I added an BUFF data example, ground truth, single view point cloud, and it's reconstruction by IF-Net)

BUFF_Example.zip

Regarding rotation, yes the network was mainly trained with forward facing scans, not sides. If that is your usecase you might want to retrain the network, including more data of persons from the side. However, the pretrained model should also do something even with a side view.

Your results do not look too wrong. The missing details might come from your input data itself - it looks like there is only very limited detail present in your point cloud - at least when looking at your images.

Hope that helps!

Best, Julian

[kfarivar]

Thank you for your response.

Can I ask what resolution you used when voxelizing the input (I guess 256 from the file name) and what was your output resolution ? since your output is visibly much sharper than mine.

[Bingoang]

@jchibane Hi, Julian~ Thanks very much for your preprocessing script! As you said, it goes from a full mesh, to a partial 3D point cloud. So it works on models with vertexes and faces, just like BUFF models. BUT when I try to use the preprocessing script to preprocess the kinect data(which is just with vertexes but no faces), it goes wrong. The error is as follows:

So, how do we preprocess the data just have vertexes but no faces(like kinect data)? Hope you can give us some advice! Thanks in advance ~ :-D

[jchibane]

Hi all,

@Bingoang In the file github_example_3DSV_creation_BUFF.py when using the voxelized_pointcloud_sampling method, use your own point cloud instead of doing point_cloud = get_3DSV(path). Make sure it is scaled and translated according to the example mesh and point cloud I provided.

@kfarivar Output resolutions of 256**3 are enough to have a high amount of detail with IF-Nets. Again: Your results are probably not limited by output resolution, but by the input. IF-Net retain details when present in the input - your data seems not to have more details. Our data was captured in a studio and naturally has a lot of details in the observed part.

[Bingoang]

Hi all,

@Bingoang In the file github_example_3DSV_creation_BUFF.py when using the voxelized_pointcloud_sampling method, use your own point cloud instead of doing point_cloud = get_3DSV(path). Make sure it is scaled and translated according to the example mesh and point cloud I provided.

@kfarivar Output resolutions of 256**3 are enough to have a high amount of detail with IF-Nets. Again: Your results are probably not limited by output resolution, but by the input. IF-Net retain details when present in the input - your data seems not to have more details. Our data was captured in a studio and naturally has a lot of details in the observed part.

@jchibane Hi, Julian! I followed your advice, and I've tried the following 2 changes: I changed point_cloud = get_3DSV(path) to point_cloud = path or point_cloud = trimesh.load(path, process=False),

but all of the attempts failed: (1)when using point_cloud = path, the error isValueError: could not convert string to float: '/home/ang/if-net-master/BUFF_Example/ang_data/yao2/yao2.ply'

(2)when using point_cloud = trimesh.load(path, process=False), the error is point_cloud = trimesh.load(path, process=False)

I guess maybe I have to preproccess to my own point cloud file before I put it into the code trimesh.Trimesh(vertices=point_cloud, faces=[]).export(out_off). But I don't know how to do, could you give me some advice? Thanks very much~

[jchibane]

Hi @Bingoang,

if I see it correctly, you do something repetitive in your code: you load the point cloud twice, once in line 63 but also in 65. AFAIK trimesh.Trimesh() expects something as a np.array, but you provide already a Trimesh object.

For such problems please refer to the corresponding documentations of the libraries (e.g. Trimesh in this case).

Best, Julian

[QhelDIV]

The file input_soccerc_hips.000128_3DSV_H250_W250_res256.npz seems corrupted, when using np.load it says: "'NpzFile' object has no attribute 'zip'". I can use 7-Zip to open the npz and see indisde, however when I opened compressed_occupancies.npy it says "CRC failed". Do I have to re-run the generation script to get over this problem?

[QhelDIV]

I've unzipped and re-zipped the npz with 7-Zip, then I can use np.load without problem.

[Robin-Mo]

Hi,thank you for your great work How to do use BUFF_Examples.zip,from input(singe view point cloud) to out(full mesh,. off file ) Thank you very much

__