training epochs and Matterport3d

[EchoTHChen]

How many epochs do you train the depth network and inpainting network respectively? And could you also release the matterport3d dataset?

[dli7319]

The number of epochs depends on the distance since the dataloader samples from the generated CARLA images. I just stopped training after 48 hours which is around 130k iterations.

For matterport3d, I used habitat-lab to generate images the same way the SynSin repo does. We did not create a new dataset for it. You'll need to download it and edit opts.scenes_dir = "../../matterport3d/mp3d_habitat" in data_readers/habitat_data.py

[EchoTHChen]

thanks

---Original--- From: "David @.> Date: Fri, Jul 22, 2022 23:10 PM To: @.>; Cc: @.**@.>; Subject: Re: [AugmentariumLab/omnisyn] training epochs and Matterport3d (Issue#3)

The number of epochs depends on the distance since the dataloader samples from the generated CARLA images. I just stopped training after 48 hours which is around 130k iterations.

For matterport3d, I used habitat-lab to generate images the same way the SynSin repo does. We did not create a new dataset for it. You'll need to download it and edit opts.scenes_dir = "../../matterport3d/mp3d_habitat" in data_readers/habitat_data.py

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[EchoTHChen]

Could you offer the config files of depth_config.txt and inpainting_config.txt for matterport3d dataset?

[dli7319]

Just added them. Also, I was using habitat-sim 0.1.6 so you may need to reinstall the right version.

[EchoTHChen]

thanks

---Original--- From: "David @.> Date: Wed, Jul 27, 2022 21:06 PM To: @.>; Cc: @.>;"State @.>; Subject: Re: [AugmentariumLab/omnisyn] training epochs and Matterport3d (Issue#3)

Just added them. Also, I was using habitat-sim 0.1.6 so you may need to reinstall the right version.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>

[EchoTHChen]

Sorry to bother you again. Does the inpainting network train for 130k iterations also? I find it is much slower than depth estimation network. The inpainting network runs about 3200 iterations for one day.(the depth estimation network runs 70k iterations.)　How can I speed up the training if I want to train inpainting network for 130k iterations with only one GPU?

[EchoTHChen]

Or how can I revise all the code to multi-gpu training?

[dli7319]

For carla, I trained ~130k iterations for both depth and inpainting. For matterport, I trained 90k for depth and 53k for inpainting. Matterport is slower so I trained inpainting for 65 hours but I still got ~20k iterations per day. The rest are trained for 48 hours exactly. All of my models are trained on one GPU so I'm not sure why it runs at only 3200 iterations per day for you. If it helps at all, I've uploaded my trained models here. Let me know if these work for you.

[EchoTHChen]

I find that it is because I set bin_size=0 for the parameters of RasterizationSetting in the MeshRasterizer. I revise it to the default parameter. And the running speed became as fast as you said. Thanks for you reply!