DifferentiableUniverseInitiative / GalFlow

A TensorFlow reimplementation of GalSim
MIT License
10 stars 4 forks source link

Add generic tools for image resampling #10

Open EiffL opened 3 years ago

EiffL commented 3 years ago

The most fundamental operation we need, besides the FFT, is the ability to do image resampling, this is crucial to many tasks from convolution to shearing.

Right now, we are relying on existing TF tools. Namely, for the x_interpolant in our convolution tools we are using the built-in tf.image.resize, this has a lot of limitations, and in particular makes weird assumptions about pixel centers, which can induce shifts. We also use the tensorflow addons image resample tool, but this one is limited to linear interpolation.

So... we want a generic tool that can resample an image using linear, cubic, quintic and lanczos kernels, and this needs to be differentiable with respect to all the inputs, including the resampling grid.

EiffL commented 3 years ago

Hummmm did a bunch of experimentation... I don't know if I can come up with an efficient implementation at the TF level for different interpolation kerrnels. Here is a notebook with some experiments for cubic interpolation, but it's super slow: https://colab.research.google.com/drive/14Jr01GgaVwkmS5ZV6RC4pjmslokgDqK_?usp=sharing

One alternative.... would be to modify the tfa implementation to support additional interpolation kernels.... would require a tiny bit of effort ^^"

EiffL commented 3 years ago

After further thought, shouldn't be impossible to do to add a higher order interpolation kernel given that pretty much all utilities for that are already in place in TFA. So, I'm opening a fork: https://github.com/DifferentiableUniverseInitiative/addons

And will be extending the resampler ( https://github.com/DifferentiableUniverseInitiative/addons/blob/master/tensorflow_addons/custom_ops/image/cc/kernels/resampler_ops.cc ) to support additional interpolation methods

EiffL commented 3 years ago

good news, cubic interpolant implemented, and 2 orders of magnitude in reduction of residuals.

Before: image

After: image