Use Torch Hub format, GIMP handling refactor, clarify license and reference details

[valgur]

Hi! This PR introduces three significant overall changes.

Using Torch Hub for model management

The current approach to providing models and sharing their weights kind of works but has some serious limitations:

• Adding new models is cumbersome. You would have to re-upload the full weights archive file and the users would have to re-download it in full accordingly.
• The user has to download the full gigabytes of weights even if some models are not relevant to their needs.
• The licenses for the models are often quite restrictive and bundling the model sources in the GIMP-ML repo makes it difficult to correctly apply a single license to the repo (such as MIT) without clashing with the model licenses.

With that in mind, PyTorch offers a convenient and consistent way to publish and use existing models + weights via their torch.hub format (with corresponding central PyTorch Hub). Loading a model is as easy as doing

model = torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', pretrained=True)

Unfortunately, publishing models in this format is not too popular yet and is quite underutilized. This can be worked around quite simply, though, by forking any interesting repo to refactor the code as needed, add hubconf.py and re-publish the model weights in a more suitable format as a release under the forked repo.

So, I did just that for all of the models used by GIMP-ML:

• valgur/DeblurGANv2, diff, weights

  predictor = torch.hub.load('valgur/DeblurGANv2:python2', 'predictor', 'fpn_inception', device=device)

• valgur/neural-colorization, diff, weights

  G = torch.hub.load('valgur/neural-colorization:pytorch', 'generator', pretrained=True, map_location=device)

• valgur/monodepth2, diff

  repo = "valgur/monodepth2"
  pretrained_model = "mono+stereo_640x192"
  encoder = torch.hub.load(repo, "ResnetEncoder", pretrained_model, map_location=device)
  depth_decoder = torch.hub.load(repo, "DepthDecoder", pretrained_model, map_location=device)

• valgur/face-parsing.PyTorch, diff, weights

  net = torch.hub.load('valgur/face-parsing.PyTorch', 'BiSeNet', pretrained=True, map_location=device)

• valgur/CelebAMask-HQ, diff, weights

  model = torch.hub.load('valgur/CelebAMask-HQ', 'Pix2PixHD', pretrained=True, map_location=device, gpu_ids=gpu_ids)

• and deeplabv3 was already availale on PyTorch Hub

  model = torch.hub.load('pytorch/vision', 'deeplabv3_resnet101', pretrained=True)

With this, the bundled model source code dirs, downloading weights.zip and using moveWeights.sh is no longer needed. Instead, the models are downloaded on the fly as necessary and with no work required by the user. I also added a hook to tqdm to get a nice indication of weight download progress directly in GIMP:

So, this solves all of the mentioned issues and more:

• The user only has to download the models they need and the process is convenient.
• Adding new models is easy.
• Clean and consistent approach to hosting the model code or weights.
• The work needed to make any existing model code portable and re-usable is not isolated to only GIMP-ML and is readily available for use by the wider PyTorch community and ecosystem instead.
• The model weights are stored in a general PyTorch cache dir and can can be easily re-used by other applications, if necessary.

Along with these changes I also improved and cleaned up the input/output handling code in the plugin scripts somewhat. Notably:

• deeplab_v3 now outputs labels with the standard color map.
• got rid of unnecessary dependencies, such as scipy, skimage and opencv.
• with the exception of facegen, all of the models now support CPU inference correctly.
• added @torch.no_grad() decorator to disable unnecessary autograd work and to avoid the need for doing .detach() on Tensors.
• got rid of the misleading scale parameter for the super resolution plugin (fixes #5)

Refactoring of GIMP wrapper code

I consolidated the GIMP<->NumPy wrapper code to shared functions under _util.py and added correct handling of layers with alpha channels. Also got rid of from gimpfu import * as it was fragile and caused frequent clashes between PIL.Image and gimpfu.Image.

Alpha channels are now removed from input images for processing and re-applied to the result automatically. The plugin input format types are now also correctly specified as RGB* or GRAY* accordingly to avoid confusing silent errors for the users. Fixes #8.

License, authors info and improved model references

The repo is in sore need of a license (as also mentioned in #11). The lack of it prevents any serious re-use, extension and collaboration on the project. And I personally like the idea behind this project a lot and would like to see it flourish and the number and variety of included models grow.

With the individual models now residing in separate repos with this PR, applying a license to the repo becomes straightforward. I propose adding a permissive MIT license and I already included one in this PR. Any changes in this PR are also licensed accordingly.

With the considerable input from this PR in mind, which is closer to a re-write than a patch, really, I also added myself as one of the contributors to the project in the readme. I hope you find it fair. I would also like to continue the collaboration in the future as well anyway, particularly by adding support for even more DL models now that it is much simpler. Besides that, the contributor credits are needed by nearly all open-source licenses as well.

I also extended and re-organized the model references somewhat. The most notable difference is again the addition of license details, which is a quite important detail for end-users since many of the models only allow non-commercial or research-only use.

If you have any comments or would like to approach something differently, let me know. I am open to any discussion.

[kritiksoman]

Sorry we were not accepting PR as GIMP was transitioning to python3 and python2 was already deprecated. If you are interested, please contribute to this branch which is meant for python 3.