Open EiffL opened 3 years ago
Hummmm did a bunch of experimentation... I don't know if I can come up with an efficient implementation at the TF level for different interpolation kerrnels. Here is a notebook with some experiments for cubic interpolation, but it's super slow: https://colab.research.google.com/drive/14Jr01GgaVwkmS5ZV6RC4pjmslokgDqK_?usp=sharing
One alternative.... would be to modify the tfa implementation to support additional interpolation kernels.... would require a tiny bit of effort ^^"
After further thought, shouldn't be impossible to do to add a higher order interpolation kernel given that pretty much all utilities for that are already in place in TFA. So, I'm opening a fork: https://github.com/DifferentiableUniverseInitiative/addons
And will be extending the resampler ( https://github.com/DifferentiableUniverseInitiative/addons/blob/master/tensorflow_addons/custom_ops/image/cc/kernels/resampler_ops.cc ) to support additional interpolation methods
good news, cubic interpolant implemented, and 2 orders of magnitude in reduction of residuals.
Before:
After:
The most fundamental operation we need, besides the FFT, is the ability to do image resampling, this is crucial to many tasks from convolution to shearing.
Right now, we are relying on existing TF tools. Namely, for the x_interpolant in our convolution tools we are using the built-in tf.image.resize, this has a lot of limitations, and in particular makes weird assumptions about pixel centers, which can induce shifts. We also use the tensorflow addons image resample tool, but this one is limited to linear interpolation.
So... we want a generic tool that can resample an image using linear, cubic, quintic and lanczos kernels, and this needs to be differentiable with respect to all the inputs, including the resampling grid.