Integrating Kornia batch augmentation pipeline.

[shijianjian]

Hi mmdet maintainers,

Thanks for the repo. We are the project lead of Kornia, and thinking of the possibilities of integrating Kornia augmentation pipeline into mmdet.

From the feature-level, we can provide:

• GPU/CPU/TPU-based batch data augmentation.
• More augmentation options, like PatchSequential.
• Pytorch-native deployable image processing using torch.jit / onnx.
• Differentiability, which has already been adopted in applications like in GANs.

Motivation: mmdet data pipeline happened outside the PyTorch computation graph and computed on CPUs in a image-by-image manner, which might increase the deployment difficulties and time legacy whilst processing large batches. With Kornia, it is much easier to include test time image processing pipelines into the model whilst export and deployment. Also, the image preprocessing could enjoy the GPU accerlations provided by PyTorch.

Our proposal:

• As demonstrated above, our whole pipeline happens differently than common CPU-based augmentations. Thus, we probably need to introduce a batch_pipeline module for such operations. Commonly:

  kornia_aug = ...
  for image, label in dataloaders:
  image = kornia_aug(image)
  logits = model(image)

  The user interface shall be like:

  model = dict(
  preprocessing=dict(
      dict(type="RandomHorizontalFlip", p=0.5),
      dict(type="ColorJitter", hue=0.1, saturation=0.3),
  ),
  backbone=dict(
      depth=101,
      init_cfg=dict(
          type='Pretrained',
          checkpoint='open-mmlab://detectron2/resnet101_caffe')))

• we might include the test-time data processing into the exported model as well.

[rbavery]

Hi! I'm a newcomer to mmdetection. Just want to say that this sounds like a really useful feature.

I haven't seen many articles on benchmarks of cpu vs gpu augmentation, but this article seems to indicate that some standard gpu augmentations tend to outperform cpu augmentations for larger image inputs: https://arxiv.org/pdf/2011.09832v1.pdf

This other article tested more augmentations without kornia, finding a similar result, though some augs performed better on the cpu: https://www.loka.com/blog/gpu-image-benchmarking

If there are more comprehensive benchmarks from Kornia comparing gpu-based augs to cpu augs, that might help drive discussion about the value of supporting kornia in mmdet? Are the maintainers of mmdet interested in a PR supporting Kornia?

[shijianjian]

Hi @rbavery, Thanks for sharing.

Those numbers in that preprint is outdated and Kornia DDA shall be faster now. I will probably start benchmarking/testing on the following in the following weeks:

• augmentation speed.
• ability to convert to JIT.
• ability to convert to ONNX.

[shijianjian]

Please check this PR for the augmentation speed accelerated brought by Kornia.

https://github.com/kornia/kornia/pull/1459#issue-781653916

[willcray]

This would be particularly useful for augmentations like CachedMosaic, which is used in the RTMDet training pipeline