Say you have a Signal represented on a discrete grid and you want to use it to learn an implicit continuous and differentiable representation of it
You can use neural networks for this purpose since they represent continuous and differentiable functions and since its representation is not directly explainable then we can refer to it as an implicit representation
Practically, you can think at the input image as a discrete mapping between
the pixel coordinates
the RGB color
You can then use that single image as a Training Set for a NN to learn this mapping
If you do so, the NN representation, contrary to the original grid based representation has the following advantages
it is differentiable
it could be more memory efficient, preserving the details
How can we do that
To test this idea, we can start with a simple NN so let's start asking ourselves
Can a simple MLP with ReLU activations learn an image?
Before doing any experiment we immediately see a fundamental limitation in this kind of NN: the ReLU activation
It is (almost) everywhere differentiable (since the NN uses it to learn by backpropagating the gradient) however its derivatives define its representation capability for high frequency and since
its first derivative is 0 or 1 and
its second derivative is 0 (almost) everywhere
then we immediately see it can't represent signals where a significant part of the information is in the higher derivatives
This equates to applying a low pass filter to the image and in fact, this is confirmed empirically: see Fig.1 Col 2
So we can try to use more nonlinear activation functions like tanh and softplus having non-zero higher-order derivatives (letting aside the Vanishing and Exploding Gradient challenges in training them) however here we can see the experimental results, even if better than the ReLU, are not that great
The main contribution of the paper is then to propose an activation function allowing to learn an implicit neural representation of an image better than any other activation considered
Overview
Paper overview
Implicit Neural Representations with Periodic Activation Functions
Arxiv: https://arxiv.org/abs/2006.09661 Project Web: https://vsitzmann.github.io/siren/
Images as NN
Say you have a Signal represented on a discrete grid and you want to use it to learn an implicit continuous and differentiable representation of it
You can use neural networks for this purpose since they represent continuous and differentiable functions and since its representation is not directly explainable then we can refer to it as an implicit representation
Practically, you can think at the input image as a discrete mapping between
You can then use that single image as a Training Set for a NN to learn this mapping
If you do so, the NN representation, contrary to the original grid based representation has the following advantages
How can we do that
To test this idea, we can start with a simple NN so let's start asking ourselves
Can a simple MLP with ReLU activations learn an image?
Before doing any experiment we immediately see a fundamental limitation in this kind of NN: the ReLU activation
It is (almost) everywhere differentiable (since the NN uses it to learn by backpropagating the gradient) however its derivatives define its representation capability for high frequency and since
then we immediately see it can't represent signals where a significant part of the information is in the higher derivatives
This equates to applying a low pass filter to the image and in fact, this is confirmed empirically: see Fig.1 Col 2
So we can try to use more nonlinear activation functions like tanh and softplus having non-zero higher-order derivatives (letting aside the Vanishing and Exploding Gradient challenges in training them) however here we can see the experimental results, even if better than the ReLU, are not that great
The main contribution of the paper is then to propose an activation function allowing to learn an implicit neural representation of an image better than any other activation considered