How long does it take to train the networks?

[hyBlue]

Hi, thanks for your code and paper. I want to ask you one thing. How many days does it take for training pairnet and fusionnet? In the supp., you trained pairnet for 600K interations with batch size of 14 and fusionnet for more than 1000K. It seems that this requires a too many days with 1 GTX1080ti gpu( mentioned in the paper).

[hyBlue]

Also when training pairnet, you trained w/o feature extractor for the first 200K and end-to-end for the rest 400K iterations. Is there any difference between your strategy and end-to-end from the beginning?

[ardaduz]

For pairnet training: Since feature extractor (MnasNet) is pretrained on ImageNet, we do not suggest and we have not experimented with unfreezing all the weights from the beginning.

For fusionnet training: We use an early checkpoint (end of first epoch, i.e. after ~100K iterations) of the pairnet modules to initialize the feature pyramid and the encoder during our experiments. This is to reduce the convergence and experimentation time. But we know for sure that it possible to train the fusionnet on ScanNet by only loading the MnasNet layer weights (freezing them for the first couple of epochs) and then training the whole model up to 1000K iterations.

Training time on 1080Ti: Training the pairnet takes around 5 days. Training the fusionnet (and then finetuning the cell) takes around 8 days. Note that if your goal is to train only the fusionnet with the described training procedure, there is no need to train the pairnet till the end, acquiring the first checkpoint is enough. As I state above, you can also try training the fusionnet with only MnasNet initialization which may converge slower but it should give similar results.

[hyBlue]

Thanks for your help!