Evaluation error

[howard84527]

Hi, I have trained the b_plane model but I got some errors when I ran " python iou.py " and " python chamfer.py "

Here is the detailed description of the error. Running " python iou.py " :

Creating filled voxels | | 1/803Traceback (most recent call last): File "iou.py", line 32, in args.overwrite) File "iou.py", line 13, in create_and_report iou.report_iou_average(kwargs) File "/home/howard/TF_14/FFD-template/template_ffd/eval/iou.py", line 129, in report_iou_average filled=filled) as ds: File "/home/howard/TF_14/FFD-template/template_ffd/eval/iou.py", line 123, in get_iou_dataset manager.save_all() File "/home/howard/TF_14/FFD-template/dids/auto_save.py", line 77, in save_all with self.get_auto_saving_dataset('a') as ds: File "/home/howard/TF_14/FFD-template/dids/auto_save.py", line 69, in get_auto_saving_dataset self.get_lazy_dataset(), File "/home/howard/TF_14/FFD-template/template_ffd/eval/iou.py", line 92, in get_lazy_dataset filled=self._filled, example_ids=example_ids) File "/home/howard/TF_14/FFD-template/template_ffd/inference/voxels.py", line 31, in get_voxel_dataset create_voxel_data(example_ids=example_ids, overwrite=False, kwargs) File "/home/howard/TF_14/FFD-template/template_ffd/inference/voxels.py", line 104, in create_voxel_data _create_filled_voxel_data(**kwargs) File "/home/howard/TF_14/FFD-template/template_ffd/inference/voxels.py", line 90, in _create_filled_voxel_data unfilled, dst, message=message, overwrite=overwrite) File "/home/howard/TF_14/FFD-template/template_ffd/data/voxels.py", line 23, in create_filled_data dst.save_dataset(src, overwrite=overwrite, message=message) File "/home/howard/TF_14/FFD-template/dids/core.py", line 204, in save_dataset value = dataset[key] File "/home/howard/TF_14/FFD-template/dids/core.py", line 477, in getitem return self._map_fn(self._base[key]) File "/home/howard/TF_14/FFD-template/template_ffd/data/voxels.py", line 19, in map_fn voxels.axis_order) AttributeError: 'RleVoxels' object has no attribute 'axis_order'

Running " python chamfer.py ":

RuntimeWarning: Mean of empty slice. out=out, **kwargs) /home/howard/TF_14/local/lib/python2.7/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount)

[howard84527]

I got the value: 0.31559994026133126. Thank you very much.

Here are some questions I want to ask you.

1. Are the 4045 training data including 803 evaluating data? I ran "scripts/vis_predictions.py" and visualized the three pics for each plane ( input template, deformed template and ground truth template). I want to know why the 803 evaluating data have the ground truth template and then drive the input template to deform?

2. Can you tell me how to prepare my own data to train? I am doing a project in my lab now. In brief, I want to deform hand and let its shape similar to the ground truth. I have the training and label data. I also render my data with eight different views, which is like your method in data pre-process. Should I continue to generate mesh, point cloud and voxel data? I hope you can give me some suggestions.

Thank you for your help.

[jackd]

1. I can't remembed exact numbers, but the training and evaluation datasets are completely separate. Ground truth CAD models exist for all. I do not understand your question.

2. You'll need to re-implement get_ffd_data and get_inputs onf TemplateFfdBuilder class.

   • get_ffd_data should return an iterable where each entry is a tuple of (cat_id, example_id, b, p). Only the b and p are used for the training. The order should be deterministic, i.e. you should get the same order of template b/p each time you run it. Look in the ffd subdirectory for code to calculate it from a poitn cloud.
   • get_inputs should return features, cloud where features is a dict containing at least the image (keyed by 'image'). Most of the code used to generate predictions etc. will probably assume it also has an exampld_id and cat_id key, but you'll have to re-implement the prediction stuff anyway.

[howard84527]

Thank you for your reply. Do you mean that I have the parameter of trained weight and I use it to deform the evaluation input model are similar to the ground truth model ? Where can I see the ground truth dataset for train and evaluation?

Thank you.

[jackd]

The trained model infers deformation parameters for specific template meshes from an image. If your image is of a plane from the evaluation dataset, hopefully the deformed template mesh will be close to the ground truth mesh.

To check out the dataset, see shapenet/examples/core/ scripts. Note I use a random test/evaluation split for training in this repository (I didn't realize an official training/evaluation split was provided) - see template_ffd/data/_ids/s0-0.8/.

[howard84527]

Understand. Thank you very much. I have other questions and want to ask you. Why the r192-256_8/my_cat_id was empty when I ran the " python render_cat.py my_cat_id -f " ? My data are .stl fies, and I import its into meshlab and export .obj files. I import the .obj files into blender because I need to zoom out the file's scale, and then exporting obj files again.

The result is below:

Rendering 13 images for cat 12345678 |██▌ | 1/13 2018-08-13 16:38:36.302849 Rendering |█████ | 2/13 2018-08-13 16:38:38.868269 Rendering |███████▍ | 3/13 2018-08-13 16:38:43.261389 Rendering |█████████▉ | 4/13 2018-08-13 16:38:47.628079 Rendering |████████████▎ | 5/13 2018-08-13 16:38:52.014676 Rendering |██████████████▊ | 6/13 2018-08-13 16:38:56.389562 Rendering |█████████████████▎ | 7/13 2018-08-13 16:39:00.770098 Rendering |███████████████████▊ | 8/13 2018-08-13 16:39:05.157176 Rendering |██████████████████████▏ | 9/13 2018-08-13 16:39:09.512153 Rendering |████████████████████████▋ | 10/13 2018-08-13 16:39:13.902825 Rendering |███████████████████████████ | 11/13 2018-08-13 16:39:18.327006 Rendering |█████████████████████████████▌ | 12/13 2018-08-13 16:39:22.712578 Rendering |████████████████████████████████| 13/13 2018-08-13 16:39:27.096405 Rendering

Thank you.

[jackd]

The rendering in shapenet is just a wrapper around this repo, with loading functions for the shapenet data specifically. If you want to render your own dataset I suspect you'll have a lot more success doing your own wrapping rather than trying to modify mine.

render_cat.py is just a bunch of loops around render_obj, which itself is just a subprocess call to to blender using blender_render.py from the other repo.

[howard84527]

I can use the stanford's repo to render my own data successfully and create eight different views for .png images, but I get wrong hdf5 file when I run "python generate_mesh_data.py " in mesh/scripts folder. I read the created hdf5 file but the key is empty.

[jackd]

... generate_mesh_data has nothing to do with renderings. It just parses the obj files and extracts/saves the vertices/faces in hdf5.

[howard84527]

Yes, I know. Can I think that the " render_cat.py " and " create_archive " only do the process which create eight different views ,png image and choose the 225° image as the input template ?

Thank you.

[jackd]

render_cat and create_archive result in an archive with 8 different views. Early versions of the code used a single view, though it has since been extended to allow for different views. You can train your network however you want - all this repo needs is a get_inputs function and some template data to deform.

[howard84527]

Hi, I can successfully render my own data. I created the mesh, point cloud and voxel data for my training and labeling data separately. Should I wrap the training and labeling data in the same hdf5 file? To import my data, I only come up with the method as the following : h = h5py.File(file_name) train_data = h['train'][:] label_data = h['label'][:]

and then using the placeholder to train the network. Is the method feasible?

Thank you.

[jackd]

1 file vs 2 is rather irrelevant - hdf5 only loads the parts you want, so it's more a personal preference thing. I normally do 1 file, then the particular train/test split is independent of the data structure on disk... but then you have to save the train/test split somewhere anyway, so there are arguments for both.

Placeholder feeding is feasible, though far from optimal. Placeholders are generally a bad idea for anything apart from simple debugging. They're slow, they don't allow easy parallelization (i.e. you can't be preparing the next batch of data while the previous batch is training) and they're not supported by tf.estimator.Estimators and many other training/evaluation frameworks.

On the other hand, the tf.data API was introduced in 1.4 and is, in my opinion, brilliant. It can be a bit intimidating at first, but once you get used to it there's not much it can't do. See my answer here for a rundown of your options. I'd go with the "wrap an index-based python function" option (though if you're using hdf5, your original dataset can be based on a list of string keys rather than a range index for simplicity).

I'd also strongly encourage you to take 15min to read over this tensorflow article on the same topic.

[af6140]

Reading the conversation here, and solving my own issues at the same time. I am wondering maybe we can use some other tool to make our code/results reproducible more reliably. For example, with docker/nvidia-docker, a clean/pristine code execution environment can be established, by publishing the docker file and related code as a whole, other interesting parties can experiment the code/research relatively easier.

For example, people are talking about it: http://tiffanytimbers.com/Using-docker-for-reproducible-research/

[jackd]

I'm sure there are better ways, but most of the issues here were (a) initial code bugs, (b) inconsistent git versions, and (c) extending code to different datasets. I'm not saying there's not a better way of doing things, but docker wouldn't solve (a) or (c), and (b) wouldn't be an issue if I used proper packaging/versioning.

I certainly learn things from experiences like this, but alas there currently isn't much motivation to make code abundantly reproducible. Peer reviews are generally anonymous and won't be able to consider code. Easily reproducible code will likely get cited more once its published, but this submission isn't at that stage yet. I publish code more in the interest of transparency than easy reproducibility. I'm happy with people taking chunks out and retro-fitting their own models (and I should have written this project more modularly - again, learning) with the ideas discussed, but I'd much rather explore new ideas (and am more incentivised in research to do so) than perfect unpublished projects on the off chance people stumble across them (not to say I'm not hapy to assist with issues as they pop up here).

[howard84527]

Hi, Can you tell me where are the label data? There are only train, test and val data in ' split.csv '. I am confused whether I should wrap my training data(obj), training label(obj) in the same hdf5 file and testing data(obj), testing label(obj) in another hdf5 file.

Thank you.

[jackd]

Point clouds are from shapenet repository - you should find the data in shapenet/core/point_clouds/_point_clouds if you've managed to train successfully.

[howard84527]

Sorry I mean I have the training data and its relative ground truth data. Should I wrap my training data and ground truth data in the same hdf5 file and then loading the hdf5 file to train?

My final goal is deforming the training data to close the ground truth data, which maybe is supervised train.

Thank you.

[jackd]

What do you mean by "training data and its relative ground truth data"? You need ground truth (or some approximation of it) for training.

[howard84527]

For example, A is training data, B is ground truth. Simply supervised network train for loss is (B-A) and then optimizing it. Now I have the A and B both. I want to deform A to close the B. Should I wrap A and B in the same hdf5 file to train? Maybe the content like following:

cat_id.hdf5:

(A) Key 'Train' : key 'traindata_1' key 'traindata_2' ... (B) Key 'Label' : key 'labeldata_1' key 'labeldata_2' ...

Thank you.

[jackd]

You can generate template data using scripts/create_ffd.py. This should sample your mesh S and give you a decomposition in templates/_ffd/3/sampled/16384 (if you use N=3 with 16384 sampled points).

These templates should not come from your evaluation/test sets - though can potentially come from your training set. That said, ascribing them the label "training data" is dangerous. You use the same templates for training and evaluation/testing. Your training data trains your network to deform your template data. If your network successfully warps template -> training data, the hope is it will also successfully warp template -> evaluation/test data.

How you save you data is up to you. My code requires a get_inputs function - so save it however you want so long as you can load it appropriately in get_inputs.

[howard84527]

I temporarily divide the input data and its ground truth data into two hdf5 files. Should I do the train/test split for the input data and ground truth data at first and then sample 1024 points to train? I see the split.csv and there are train/test/val. What's the difference between test and val? Are the val data used to deform?

I can use the scripts/create_ffd.py to generate my template data. I should apply the trained weights to the template data if I successfully train the network, right?

Thank you.

[jackd]

At the time of writing the paper this repo was based on I was unaware of the official train/test/split provided by shapenet core. When I became aware of it, I added it to the shapenet repo, but I already submitted the paper this repo is based on, so left this repo to use the train/eval split I used for the experiments. There is no test split for this repo.

The template data is a component of the model. Think of it as an untrained parameter of the final layer. It should remain constant across training and evaluation/prediction.