Is the the TSDF fusion steps necessary for training a model?

[YokkaBear]

Hi, thank you for your brilliant work pretty much, and I have starred your work.

Now I try to use a new 3D dataset to train a new model for a new research based on your code, and now I have got the original 2D images and 3D .off files. During I ran your code in /external/mesh-fusion to preprocess the 3D data, I met some trouble.

I have finished the 'Installation' section, however, when I dived into the 'Usage' section, I could only ran over the first command, which is used for scaling the models, and the second and third commands falied to run. I have spent some days on debugging but the bugs still exists. And I am really not willing to spend more days on it.

So, I just wonder, whether or not could I skip the TSDF preprocessing steps, and train the new model with my own dataset directly? Much appreciations if your could reply my question.

[LMescheder]

Hi @YokkaBear, that depends if your data already consists of watertight meshes. In this case, you can directly use the sample_mesh.py script. Otherwise, you have to make the meshes watertight first (that's what we are doing with meshfusion). However, there are other options, like Manifold that can also do it. If you provide more information on what you are trying to do and what errors you get, we might also be able to help you.

[YokkaBear]

Thank you for your quick reply @LMescheder , and I will give you my feedback after I go deeper into these methods.

[YokkaBear]

Hi, @LMescheder, Thank you for your kindly instructions, I have finished running the demo of Manifold and could generate simplified watertight 3D meshes by hand. My current research project is about the reconstruction of 3D human face, since I found that your model could reconstruct 3D models in ShapeNet dataset, I think your model should also be effective in the 3D face construction task. So currently the question is, now that I have been able to preprocess the 3D face models, how could I construct and organize a new trainable 3D face dataset like the ShapeNet used in your project? i.e. What structure is the new dataset supposed to have? I will appreciate a lot if you could give me some detailed advice.

[YokkaBear]

Maybe the description above is a little abstract. I will take it brief. To now I have been able to make watertight mesh models of any 3D project in the dataset, so I think I could make a dataset, where each single image is mapping to a 3D watertight mesh. (my focus is on single image reconstruction.)
My question is, how should I organize the file structure of the dataset, so that it can be trained by your training code? I will check out the build.sh and mesh_sample.py as well, but it would be better if you could show me the file structure of your dataset used for training in detail. Thanks a lot again.

[LMescheder]

Hi @YokkaBear , take a look at our processed ShapeNet data you can download. The structure is

$CLS/$MODELNAME/pointcloud.npz
$CLS/$MODELNAME/points.npz
$CLS/$MODELNAME/model.binvox
$CLS/$MODELNAME/img/%03d.png

where pointcloud.npz contains the pointcloud, points.npz contains the sampled points (in the volume) + GT occlusion, model.binvox is a voxelization (if needed) and the img folder contains the views. Note that you can change all these names by adapting the corresponding option in the config file. In addition, you can specify the dataset splits using

$CLS/$MODELNAME/{train,val,test}.lst

[YokkaBear]

Hi, @LMescheder , In order to learn more about the training process, I think it is better to run the training code with the processed dataset that you have provided (as you say, the option 1). I have two questions:

1. Is the processed dataset you provided (at the URL: https://s3.eu-central-1.amazonaws.com/avg-projects/occupancy_networks/data/dataset_small_v1.1.zip) can be directly used for train the network instead of other processing step? i.e. Does it cover pointcloud.npz/points.npz/model.binvox/img the four kinds of data as you mentioned before?
2. Could you please offer a google drive download link, since in my region my access to the amazon aws is a little unstable, and the download takes as long as a few days, and the connection always cuts down which takes longer.

Happy new year, and I'll greatly appreciate it if you could offer some help.

[YokkaBear]

https://github.com/autonomousvision/occupancy_networks/issues/27#issuecomment-570518411
Fortunately I have downloaded the dataset and try the next step to run it through, no longer is the google drive link needed.

[YokkaBear]

(This comment is a little bit long, taking 3~5 min to read, excuse me.) Hi, @LMescheder Up to now I have succeeded in training the preprocessed ShapeNet dataset, and the metrics are quite close to those in your paper. So as to use your training method on my own 3D face reconstruction project, I have got the watertight version of my own 3D face dataset, and used scripts/sample_mesh.py to obtain the pointcloud.npz, points.npz and model.binvox for each watertight object. Together with different views of the object in "img" folder, I have made a dataset in the form as the preprocessed ShapeNet. However, when I used the home-made dataset for training, the following error occurs, which blocks the training:

Error occured when loading field points of model 0019
None
Error occured when loading field points of model 0019
None
Error occured when loading field points of model 
Traceback (most recent call last):
  File "train.py", line 78, in <module>
    data_vis = next(iter(vis_loader))
  File "/root/miniconda3/envs/mesh_funcspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/root/3dface/onet/im2mesh/data/core.py", line 168, in collate_remove_none
    return data.dataloader.default_collate(batch)
  File "/root/miniconda3/envs/mesh_funcspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 200, in default_collate
    elem_type = type(batch[0])
IndexError: list index out of range

Those 'None' results should have showed the first element of variables 'train_dataset' and 'val_dataset' as the debugging lines that I inserted, but they didn't:

print(train_dataset[0])  # debug
print(val_dataset[0])  # debug

And this means the failure in loading the dataset. But when I put one directory (named "1011e1c9812b84d2a9ed7bb5b55809f8") from your preprocessed ShapeNet dataset to my own dataset location, the training process passes. So I suppose there must be some problems with my processed dataset. After I unzip the npz files, I found an evident difference in points/occupancies.npy. In my home-made dataset, the occupancies array looks like: array([False, False, False, ..., False, False, False]) But in your preprocessed ShapeNet, the occupancies array is like: array([ 0, 16, 0, ..., 0, 0, 0], dtype=uint8)

The element type and content are totally different. So I guess that the problem is most likely to be caused by "sample_mesh.py". And What I wonder is if any specific argument parser assignments when running the sample_mesh.py? Here I'd like to show an example of the command line input I used, which may be wrong:

python3 sample_mesh.py --in_folder /root/3dface/onet/data/examples-3dface/output --pointcloud_folder /root/3dface/onet/data/examples-3dface/pointcloud --voxels_folder /root/3dface/onet/data/examples-3dface/binvox --points_folder /root/3dface/onet/data/examples-3dface/points

Looking forward to your advice, much gratitudes.

[YokkaBear]

my last comment is solved partially by adding --packbits option to the command line instruction of running sample_mesh.py as below:

python3 sample_mesh.py --in_folder /root/3dface/onet/data/examples-3dface/output/output-off-5to17 --pointcloud_folder /root/3dface/onet/data/examples-3dface/pointcloud --points_folder /root/3dface/onet/data/examples-3dface/points --voxels_folder /root/3dface/onet/data/examples-3dface/binvox --resize --float16 --points_size 2048 --packbits

This change can fix the problem of loading training dataset (variable "train_dataset" in train.py), but the validation dataset still cannot be loaded, which prompts the following erros:

Error occured when loading field voxels of model 0019
None
Error occured when loading field voxels of model 0019
Error occured when loading field points of model 
Traceback (most recent call last):
  File "train.py", line 79, in <module>
    data_vis = next(iter(vis_loader))
  File "/root/miniconda3/envs/mesh_funcspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/root/3dface/onet/im2mesh/data/core.py", line 168, in collate_remove_none
    return data.dataloader.default_collate(batch)
  File "/root/miniconda3/envs/mesh_funcspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 200, in default_collate
    elem_type = type(batch[0])
IndexError: list index out of range

Really looking forward to your help, much thanks.

[MingwuZheng]

Hi, have u reconstructed human face using onet successfully?

[YokkaBear]

Hi, have u reconstructed human face using onet successfully?

Yes, to now I have run the training and testing process on my own 3D face dataset several times to get better results, because the IoU metric performs much too low.

[MingwuZheng]

Hi, have u reconstructed human face using onet successfully?

Yes, to now I have run the training and testing process on my own 3D face dataset several times to get better results, because the IoU metric performs much too low.

Thanks to your quick reply. Can you share more details about your face model and dataset？What parameters do I need to change？ By the way, when I performed manifold to create watertight mesh, my face mesh became topologically closed, which made it have two layers. Have you met and sloved this problem? Hoping for your reply, and, if it is convenient for you, can you provide your contact information like CSDN or emails for further communication？Much thanks！

[YokkaBear]

The model I applied now is the ONet proposed by the writer (without any modification up to now), the dataset I applied is the Florence3DFace dataset, and the parameters to change are mainly located in the .yaml files and some variable names in training/generation/testing codes.
I did not encounter the two-layer problem during watertight meshes generation, maybe it is caused by the problems of the dataset.

[MingwuZheng]

The model I applied now is the ONet proposed by the writer (without any modification up to now), the dataset I applied is the Florence3DFace dataset, and the parameters to change are mainly located in the .yaml files and some variable names in training/generation/testing codes. I did not encounter the two-layer problem during watertight meshes generation, maybe it is caused by the problems of the dataset. My email is 15640162499@163.com, welcome for communications.

Hi, I have sent you an email in Chinese. Thank you and hoping for your reply.

[ALI7861111]

my last comment is solved partially by adding --packbits option to the command line instruction of running sample_mesh.py as below:

python3 sample_mesh.py --in_folder /root/3dface/onet/data/examples-3dface/output/output-off-5to17 --pointcloud_folder /root/3dface/onet/data/examples-3dface/pointcloud --points_folder /root/3dface/onet/data/examples-3dface/points --voxels_folder /root/3dface/onet/data/examples-3dface/binvox --resize --float16 --points_size 2048 --packbits

This change can fix the problem of loading training dataset (variable "train_dataset" in train.py), but the validation dataset still cannot be loaded, which prompts the following erros:

Error occured when loading field voxels of model 0019
None
Error occured when loading field voxels of model 0019
Error occured when loading field points of model 
Traceback (most recent call last):
  File "train.py", line 79, in <module>
    data_vis = next(iter(vis_loader))
  File "/root/miniconda3/envs/mesh_funcspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/root/3dface/onet/im2mesh/data/core.py", line 168, in collate_remove_none
    return data.dataloader.default_collate(batch)
  File "/root/miniconda3/envs/mesh_funcspace/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 200, in default_collate
    elem_type = type(batch[0])
IndexError: list index out of range

Really looking forward to your help, much thanks.

DID YOU SOLVE THAT ISSUE I AM FACING THE SAME ISSUE WITH CUSTOM DATA ....??

[zyz-notebooks]

嗨， @LMescheder，非常 感谢您的指导，我已经完成了Manifold的演示，可以手动生成简化的水密3D网格。 我目前的研究项目是关于3D人脸的重建，因为我发现您的模型可以在ShapeNet数据集中重建3D模型，因此我认为您的模型在3D人脸构建任务中也应该是有效的。因此，目前的问题是，既然我已经能够预处理3D人脸模型，那么我该如何构建和组织一个新的可训练3D人脸数据集，例如您项目中使用的ShapeNet？即新数据集应该具有什么结构？如果您能给我一些详细的建议，我将不胜感激。

Hello, ./simplify -i input.obj -o output.obj -m -c 1e-2 -f 10000 -r 0.2 Segfault (core dumped)

[YokkaBear]

@zyz-1998 I didn't encounter this error, try checking whether the files exist, whether there are errors in input arguments, or in OS.

[YokkaBear]

@ALI7861111 For me, the error occurs because of the size of voxels of my dataset is not 32*32*32, and I modify the code in im2mesh/utils/binvox_rw.py to solve this.

[zyz-notebooks]

@zyz-1998 I didn't encounter this error, try checking whether the files exist, whether there are errors in input arguments, or in OS.

when I run python train.py configs/img/onet.yaml , Do you have a solution RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 5.80 GiB total capacity; 4.42 GiB already allocated; 32.81 MiB free; 71.76 MiB cached)

[YokkaBear]

@zyz-1998 This indicates that the GPU memory that your program applies to use surpasses the maximum size of your GPU card. Try switching another available GPU by resetting the column "CUDA_VISIBLE_DEVICES".

[zyz-notebooks]

@zyz-1998 This indicates that the GPU memory that your program applies to use surpasses the maximum size of your GPU card. Try switching another available GPU by resetting the column "CUDA_VISIBLE_DEVICES".

Thank you very much for your answer. I have solved it by adjusting the size of batch_size. I still have some problems about the self-made data set. I sent an email in Chinese to your 15640162499@163.com and look forward to your reply.

[csyhping]

@ALI7861111 For me, the error occurs because of the size of voxels of my dataset is not 323232, and I modify the code in im2mesh/utils/binvox_rw.py to solve this.

Hi @YokkaBear , could you please tell how did you modify in binvox_rw.py? I've met the same problem. Thanks !

[YokkaBear]

@csyhping I modify the variable 'data' in function read_as_3d_array() in im2mesh/utils/binvox_rw.py, by padding zeros until the dimension of it is 32x32x32=32768.

[csyhping]

@csyhping I modify the variable 'data' in function read_as_3d_array() in im2mesh/utils/binvox_rw.py, by padding zeros until the dimension of it is 32x32x32=32768.

Hi @YokkaBear , thanks for you quick reply! I checked my .binvox file, it seems the dim is ok, but the data is not. According to read_as_3d_array(), there should be pair of value, counts, the value should be 0/1, but in my binvox file, the value seems ranges in [0,255]. I use the sample_mesh.py to generate the binvox file, but something is wrong. Do you know how to fix this?

[YokkaBear]

Sorry maybe I didn’t meet that error or maybe I forget, since it's been a few months ago. You could try adding some exception throwing module or breaking points to check which part of code raise that wrong output.

[chunyangL7]

@csyhping I modify the variable 'data' in function read_as_3d_array() in im2mesh/utils/binvox_rw.py, by padding zeros until the dimension of it is 32x32x32=32768.

Hi @YokkaBear , thanks for you quick reply! I checked my .binvox file, it seems the dim is ok, but the data is not. According to read_as_3d_array(), there should be pair of value, counts, the value should be 0/1, but in my binvox file, the value seems ranges in [0,255]. I use the sample_mesh.py to generate the binvox file, but something is wrong. Do you know how to fix this?

Hi, I fix this by this issue, I believe it is work for you.