search

autonomousvision / differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf
MIT License
798 stars 90 forks source link

How to train the model on a single class of Shapenet Data? #14

Closed nitish11 closed 4 years ago

nitish11 commented 4 years ago

If I want to retrain the model on a single class of Shapenet Data, what are the variables to be changed in the configuration file? What is the use of pointcloud chamfer file mentioned in the config file? https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/default.yaml#L10

m-niemeyer commented 4 years ago

Hi @nitish11 , thanks a lot for your interest in our project!

Single Class Training: You can train on a single class of the ShapNet dataset by adjusting the data - classes argument in your config. For example, if you only want to train on cars which is the class "02958343", you have to set

data:
  classes: ['02958343']

What is the Pointcloud Chamfer file? For evaluating our and baseline methods, we report the Chamfer distance. The pointcloud chamfer file entry in the config defines the name of the file containing the ground truth points which can be found in the respective model folder. We load the respective field with this name as an argument for evaluation.

nitish11 commented 4 years ago

Hi @m-niemeyer, thanks fro your reply and clarifying the doubt.

I want to train a model on only one class,; i.e; speaker - 03691459 . Before downloading the data using your script, I just wanted to confirm if Kato shapenet data contains Speaker class or not. Could you please confirm.

m-niemeyer commented 4 years ago

@nitish11 yes, both ShapeNet datasets (Kato renderings and Choy renderings) contain the speaker class.

nitish11 commented 4 years ago

@m-niemeyer : Thanks for the confirmation, I downloaded the NMR_dataset. And copied only one folder 03691459. I modified config/default.yaml as mentioned below.

method: dvr
data:
  dataset_name: Shapes3D
  path: data/ShapeNet
  classes: ['03691459']
  input_type: image
  dim: 3
  n_views: 0
  n_views_input: 0
  pointcloud_chamfer_file: pointcloud.npz
  depth_folder: depth
  img_folder: image
  visual_hull_depth_folder: visual_hull_depth
  mask_folder: mask
  img_extension: png
  img_extension_input: jpg
  depth_extension: exr
  mask_extension: png
  img_size: null

What is the use of "img_folder_input"? https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/default.yaml#L13 ?

I am getting the error Error occurred when loading field inputs of model c3233311a94b7b968a0181490cb73cbc (03691459), the variables that are fetched at this point are as follows

extension: jpg
folder: data/ShapeNet/03691459/c3233311a94b7b968a0181490cb73cbc/img_choy2016
files: []

The dataset contains 

$ ls data/ShapeNet/03691459/c3233311a94b7b968a0181490cb73cbc
image mask visual_hull_depth cameras.npz pointcloud.npz
m-niemeyer commented 4 years ago

Hi @nitish11,

yes, you are getting the error because you are not using a correct config file. The NMR Dataset you downloaded is only used by the RGB model which can be trained using this config file.

_img_folderinput is the name of the image folder; In the NMR Dataset, it is "image".

If you want to train the RGB model with on the category "03691459" from the NMR Dataset, you can add

  classes: ['02958343']

under data of this config file, but you have to keep the other options as they are.

To be completely clear, you end up with a config file with the following config:

data:
  dataset_name: NMR
  path: data/NMR_Dataset
  train_split: softras_train
  val_split: softras_val
  test_split: softras_test
  img_folder_input: image
  img_extension_input: png
  depth_from_visual_hull: true
  depth_range: [0., 5.]
  classes: ['02958343']
model:
  decoder_kwargs:
    hidden_size: 128
training:
  out_dir:  out/single_view_reconstruction/multi_view_supervision/ours_rgb
  n_workers: 8
  model_selection_metric: mask_intersection
  model_selection_mode: maximize
  n_eval_points: 4096

Please check this section of our readme for how to start training a model. I hope this helps!

nitish11 commented 4 years ago

@m-niemeyer : Thanks for the explanation.

Now, I am getting error in reading depth_from_visual_hull.

For filename : data/NMR_Dataset/03691459/dc9c9ce6dc9a771f500158c23c4c5a8e/visual_hull_depth/00060001.exr

I am getting below error.

in load_visual_hull_depth
    depth = np.array(imageio.imread(filename)).astype(np.float32)
  File "/home/nitish/anaconda3/envs/s3d/lib/python3.8/site-packages/imageio/core/functions.py", line 265, in imread
    reader = read(uri, format, "i", **kwargs)
  File "/home/nitish/anaconda3/envs/s3d/lib/python3.8/site-packages/imageio/core/functions.py", line 181, in get_reader
    raise ValueError(
ValueError: Could not find a format to read the specified file in single-image mode
m-niemeyer commented 4 years ago

I think this might be related to your imageio library which cannot read exr files. Can you please have a look at this issue?

In particular, I think adding

import imageio
imageio.plugins.freeimage.download()

to e.g. train.py should fix the problem!

nitish11 commented 4 years ago

Thanks for the suggestion. 

import imageio
imageio.plugins.freeimage.download()

But we need to update the above lines here in im2mesh/data/fields.py.

ujjawalcse commented 3 years ago

Hi @nitish11 @m-niemeyer , Thanks for your clear and elaborative discussion that helped me a lot.

I could be able to train the model for a custom single class using this config file - method: dvr data: dataset_name: NMR path: /home/ujjawal/my_work/object_recon/differentiable_volumetric_rendering/data/j_data classes: ['00'] input_type: image dim: 3 n_views: 0 n_views_input: 0 pointcloud_chamfer_file: pointcloud.npz depth_folder: depth img_folder: images img_folder_input: images visual_hull_depth_folder: depth mask_folder: mask img_extension: png img_extension_input: png depth_extension: png mask_extension: png img_size: null img_size_input: 224 img_with_camera: true img_with_mask: true train_split: train val_split: val test_split: test cache_fields: false split_model_for_images: false depth_from_visual_hull: false depth_range: [0, 2.4] ignore_image_idx: [] model: decoder: simple encoder: resnet18 decoder_kwargs: {} encoder_kwargs: {} depth_function_kwargs: {} c_dim: 256 lambda_occupied: 1. lambda_freespace: 1. lambda_rgb: 1. lambda_depth: 0. lambda_image_gradients: 0. lambda_sparse_depth: 0. lambda_normal: 0.05 reduction_method: sum patch_size: 1 training: out_dir: out/00 batch_size: 2 batch_size_val: 1 print_every: 10 checkpoint_every: 200 visualize_every: 10000 validate_every: 2000 backup_every: 50000 learning_rate: 0.0001 scheduler_milestones: [750, 1000] scheduler_gamma: 0.5 model_selection_metric: loss model_selection_mode: minimize n_training_points: 4 n_eval_points: 4 n_workers: 0 logfile: train.log sample_continuous: False overwrite_visualization: true occupancy_random_normal: False depth_loss_on_world_points: False use_cube_intersection: true always_freespace: True multi_gpu: false test: eval_file_name: eval_meshes threshold: 0.5 model_file: model_best.pt generation: batch_size: 1 vis_n_outputs: 30 generation_dir: generation simplify_nfaces: null padding: 0.0 resolution_0: 32 upsampling_steps: 3 refinement_step: 30 refine_max_faces: 10000 with_colors: true mesh_extension: ply rendering: render_dir: rendering colors: rgb resolution: [256, 256] ray_sampling_accuracy: [1024, 1025] extension: jpg n_views: 1 n_start_view: 0 background: white

  1. "images" contain multi-view images (png)
  2. "mask" contain corresponding masks of these images (png)
  3. "depth" contain corresponding depth of these images (png)
  4. "cameras.npz" contain require .mat files

So, Now I wanna know how to train this pipeline for single RGB images for different classes so that we can generate 3D model from a single image only. What changes need to be done in config file? Or Mention any other changes need to be done.

nitish11 commented 3 years ago

Please refer the below config file also and updates the parameters. https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/single_view_reconstruction/single_view_supervision/ours_combined.yaml

nitish11 commented 3 years ago

Once you have updated config file, then follow the instructions from here. https://github.com/autonomousvision/differentiable_volumetric_rendering#training

nitish11 commented 3 years ago

After training the model, then you can refer the below section for testing. https://github.com/autonomousvision/differentiable_volumetric_rendering#generation-from-your-own-single-images

ujjawalcse commented 3 years ago

Thank you very much Nitish, I'll try it out by tomorrow.

On Fri, 15 Jan 2021, 17:28 Nitish Bhardwaj, notifications@github.com wrote:

After training the model, then you refer the below section for testing.

https://github.com/autonomousvision/differentiable_volumetric_rendering#generation-from-your-own-single-images

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/autonomousvision/differentiable_volumetric_rendering/issues/14#issuecomment-760900540, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH2BJCHWC7K75IRJ4X7LZTTS2AUXRANCNFSM4NLYGK7Q .

ujjawalcse commented 3 years ago

Hi Nitish, I tried with this config https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/single_view_reconstruction/single_view_supervision/ours_combined.yaml But I'm facing many KeyError as train.py expect all the keys present in this config https://github.com/autonomousvision/differentiable_volumetric_rendering/blob/master/configs/default.yaml

Traceback (most recent call last): File "train.py", line 61, in <module> train_dataset = config.get_dataset(cfg, mode='train') File "/home/ujjawal/my_work/object_recon/differentiable_volumetric_rendering/im2mesh/config.py", line 151, in get_dataset method = cfg['method'] KeyError: 'method'

I tried adding the method parameter in the config file , but after this , again i got error for cachefields and so this continues for other parameters also . How to deal with this?