search

QUVA-Lab / e2cnn

E(2)-Equivariant CNNs Library for Pytorch
https://quva-lab.github.io/e2cnn/
Other
599 stars 74 forks source link

wrapping pytorch operations - grid_sample #47

Closed ahyunSeo closed 3 years ago

ahyunSeo commented 3 years ago

Hello,

Thank you for your nice work!

I'm about to use the e2cnn based network for my project. I need to use the method grid_sample in PyTorch and its underlying implementation is just some interpolation. https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html?highlight=grid_sample#torch.nn.functional.grid_sample I noticed the upsample in this repo simply computes the value and reassigns the output type. https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/nn/modules/r2upsampling.py#L86 I think I can do exactly like this with grid_sample. Please let me know if there will be any issues with this.

Ahyun

Gabri95 commented 3 years ago

Hi @ahyunSeo

What do you need grid-sample for?

Wrapping it in an EquivariantModule (as done in r2upsampling) is indeed not a problem from an implementation point of view. However, grid_sample can perform some arbitrary deformation of the input, which means it is not generally equivariant and, therefore, should not be implemented in a subclass of EquivariantModule.

Are you maybe assuming some specific grid which guarantees equivariance?

Best, Gabriele

ahyunSeo commented 3 years ago

Hi @Gabri95

You made a fair point. I was about to sample a circular/polar grid for each pixel, where each pixel becomes its center. So each pixel will be assigned with # angle of grid points. Then, I wanted to compute correlation values between the grid points per pixel for the further pipeline. Maybe I should move the feature tensor to a non-geometric tensor first.

Regards, Ahyun

Gabri95 commented 3 years ago

Hi @ahyunSeo ,

If the operation you are trying to implement is equivariant you can just wrap it in an EquivariantModule as I did for r2upsampling. In the forward pass, you just extract the 'tensor' from the GeometricTensor input, process this tensor is an equivariant way and finally wrap the output in a new GeometricTensor.

You can precompute the grid in the init of the module such that you can ensure the grid allows equivariance.

Let me know if this helps

Gabriele

ahyunSeo commented 3 years ago

Thanks a lot!