search

jonkhler / s2cnn

Spherical CNNs
MIT License
939 stars 176 forks source link

S2Convolution requires square inputs #27

Open meder411 opened 5 years ago

meder411 commented 5 years ago

Why does the S2Convolution layer require square inputs? Is there a way around this to explore the use of your convolutions on equirectangular images?

pochoi commented 5 years ago

GitHub Logo https://www.cs.dartmouth.edu/~geelong/sphere/

Here is a nice picture on how grid points lying on sphere. The dimension (2 b_in, 2 b_in) of the input image is the design of the grid points on the sphere. There are total 2 b_in 2 * b_in black dots in the picture.

Such grid point design is the convention of the discrete spherical harmonics transform used in this work.

meder411 commented 5 years ago

Ah, okay. So processing an equirectangular projection would require a resampling of the image?

Equirectangular images are usually (180, 360) resolution--essentially unwrapping the sphere around the vertical axis. So I guess I'd need to resample it so that the latitudinal dimension is upsampled by a factor of 2?

pochoi commented 5 years ago

Why not? In the MINST example, the projection of the 2D image to spherical image can be considered as a kind of resampling. The projection/resampling done in the MNIST example is not the only way. We should feel free to try different projection/resampling schemes to see which works better.

meder411 commented 5 years ago

Okay, thanks for the explanation! I'll explore a bit.

weberhen commented 5 years ago

I thought that the MNIST dataset is projected on the northern hemisphere only, in which case having a square image makes sense. But once one works on the full sphere, then we need to resample since the spherical projection are 180° - 360° as @meder411 said.