adynathos
commented
2 years ago Your observation is right and indeed in the current form the implementation is not real time. I used the original code for the inpainting and it has some limitation on performance:
- Tensorflow implementation, needed transferring data between back-and-forth to the GPU to work with our PyTorch
- no batch dimension, a single patch per network run
- not taking advantage of half-precision which is faster on some newer GPUs
So here we do research of the idea, rather than a system ready for production. But I believe there are many opportunities to speed up the networks for deployment:
- Re-using backbone features between parts of the pipeline.
- Re-using backbone features for inpainting patches, by doing the masking out in intermediate layers rather than repeating the feature extraction for every masked patch.
- Network distillation.
johannesTech
Hello @adynathos,
first of all, I want to thank you for sharing your great research in the published paper!
I am very interested in the proposed algorithm and started experimenting with the inpainting algorithm in particular. Running the pretrained model from J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang "Free-Form Image Inpainting with Gated Convolution" alone on image-patches of size 400x400 with an inpainting mask of size 200x200 and an overlap of 0.7 (on the masked/erased area or the patch-area?) seems to result in rather large runtimes that make me question the real-time application of the proposed algorithm. But maybe this is just a problem with my implementation of it, so I am looking forward to your code :) Did you take a look into the runtime of your algorithm and have some insight?
Thank you for your reply.
Best, Johannes