search

TRI-ML / sdflabel

Official PyTorch implementation of CVPR 2020 oral "Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors"
MIT License
161 stars 19 forks source link

Autolabeling custom 3D Objects #3

Closed happyCodingSusan closed 3 years ago

happyCodingSusan commented 3 years ago

I want use your method to label my own 3D objects (CAD models are available). What steps do i need to follow?

zakharos commented 3 years ago

Hi Susan! If you already have CAD models available and shape estimation is not really needed, then you only need to train a detector network (RCNN, DETR etc.) and a correspondence network (CSS net). The steps would be:

  1. Train a detector to estimate bounding boxes containing your objects. If you don't have an annotated training set, you could render your CAD models, e.g. using BlenderProc;
  2. Additionally render NOCS correspondences for your objects (similar to data/db/crops) e.g. using nocs-renderer;
  3. Train a correspondence net to output NOCS from RGB (similar to train_css.py);
  4. Estimate poses for detected objects based on estimated NOCS either by using PnP+RANSAC (if you have only RGB images available) or Kabsch+RANSAC (if you also have depth at hand). You can use our code for that;
  5. Additionally you could apply refinement based on RGB/Depth similar to refine_css.py.
happyCodingSusan commented 3 years ago

@zakharos ,

Thank you very much for your kind and quick reply! I tried the nocs-renderer, but encountered some runtime errors. I wonder if you can tell me all the packages and their versions in your renderer environment. I set up the conda environment on windows 10. When I run "conda env create -n renderer -f env.yml" for the first time, "glumpy" cannot be installed successfully. After I changing python=3.5 in env.yml to python=3.6 and adding cython, I can create the environment. Then I encountered a runtime error at "expm" function. After reinstalling scipy of version 1.3.0, I can run the program and 24 images [ "synth_0000(~0007)_dpt.png, synth_0000(~0007)_dpt_vis.png, synth_0000(~0007)_img.png" ] were saved in the output folder. Howerer, it crashed at "self.program.draw(output, model.index_buffer)" in the function "draw_model". I guess there are still some package versions that do not match. I copied my conda renderer env packages and versions, as shown below, would you please send me your entire renderer environment packages and versions? Thank you very much for your help, Susan.

| packages | in environment | at C:\Anaconda\envs\renderer:

| Name | Version | Build Channel

blas | 1 | mkl | anaconda certifi | 2020.12.5 | py36haa95532_0 |   cffi | 1.14.4 | py36hcd4344a_0 |   cudatoolkit | 10.1.243 | h74a9793_0 |   cython | 0.29.21 | py36ha925a31_0 | anaconda decorator | 4.0.11 | |   freetype | 2.10.4 | hd328e21_0 |   future | 0.16.0 | |   glumpy | 1.1.0 | |   icc_rt | 2019.0.0 | h0cc432a_1 | anaconda intel-openmp | 2020.2 | 254 | anaconda jpeg | 9b | hb83a4c4_2 |   libpng | 1.6.37 | h2a8f88b_0 |   libtiff | 4.1.0 | h56a325e_1 |   lz4-c | 1.9.3 | h2bbff1b_0 |   mkl | 2019.4 | 245 | anaconda mkl-service | 2.3.0 | py36hb782905_0 | anaconda mkl_fft | 1.2.0 | py36h45dec08_0 | anaconda mkl_random | 1.1.0 | py36h675688f_0 | anaconda networkx | 1.11 | |   ninja | 1.10.2 | py36h6d14046_0 |   numpy | 1.19.1 | py36h5510c5b_0 | anaconda numpy-base | 1.19.1 | py36ha3acd2a_0 | anaconda olefile | 0.46 | py_0 |   opencv-python | 3.1.0.5 | |   pillow | 8.1.0 | py36h4fa10fc_0 |   pip | 20.2.4 | py36_0 | anaconda plyfile | 0.5 | |   pycparser | 2.2 | py_2 |   pyglet | 1.2.4 | |   pyopengl | 3.1.1a1 | py36_0 | anaconda python | 3.6.12 | h5500b2f_2 | anaconda pytorch | 1.4.0 | py3.6_cuda101_cudnn7_0 | pytorch scipy | 1.3.0 | |   setuptools | 50.3.0 | py36h9490d1a_1 | anaconda six | 1.15.0 | py_0 | anaconda sqlite | 3.33.0 | h2a8f88b_0 | anaconda tk | 8.6.10 | he774522_0 |   torchvision | 0.5.0 | py36_cu101 | pytorch triangle | 20200424 | |   vc | 14.1 | h0510ff6_4 | anaconda vispy | 0.4.0 | |   vs2015_runtime | 14.16.27012 | hf0eaf9b_3 | anaconda wheel | 0.35.1 | py_0 | anaconda wincertstore | 0.2 | py36h7fe50ca_0 | anaconda xz | 5.2.5 | h62dcd97_0 |   zlib | 1.2.11 | vc14h1cdd9ab_1 | [vc14] anaconda zstd | 1.4.5 | h04227a9_0 |  

zakharos commented 3 years ago

Hi Susan,

Sorry for the late reply. Unfortunately I didn't use the NOCS renderer on Windows and I currently don't have a machine to try... However, please create an issue in the respective repository and maybe some other users (or myself once I have access to a Windows machine) will be able to provide an environment for Windows. Thanks!