Compatibility with Scale Equivariance

[ardasahiner]

Thank you for this great implementation of equivariant CNNs. I noticed that in your paper and in this implementation, while broad groups such as O(2) are mentioned, only transformations of translation/rotation/reflection are explicitly discussed.

In light of new work on scale-equivariant CNNs, such as this one, would scale-equivariance also be covered under O(2), since the origin is preserved under scaling? If so, how could scale-equivariance be incorporated in tandem with the transformations covered in this library?

I would appreciate any tips or thoughts you might have on this subject.

[ejnnr]

while broad groups such as O(2) are mentioned, only transformations of translation/rotation/reflection are explicitly discussed.

O(2) is exactly the group of rotations + reflections, it doesn't contain any other transformations. In particular, scaling is not part of O(2). All linear maps preserve the origin, the special property of O(2) is that it keeps all distances invariant (which scaling of course doesn't do).

how could scale-equivariance be incorporated in tandem with the transformations covered in this library?

The paper you linked seems to take the group-convolutional viewpoint, which would correspond to regular representations in this library. If that's what you want to do, combining scale-equivariance with rotation equivariance should be relatively straightforward in principle: for scale-equivariance, you can use a basis that contains each basis function at S different scales (like in the SENS paper), and similarly, for rotation equivariance under e.g. C_N, you can use a basis with N different rotated versions of each basis function (see e.g. this paper). To combine these, I think you could just use a basis with S x N elements, which would contain all combinations of rotations and scalings you're interested in.

But I'd guess that approach isn't a good fit for the e2cnn library because the e2cnn library supports arbitrary group representations, rather than only regular ones. And since the scaling group (i.e. the positive real numbers) is non-compact, I think finding the complete basis for arbitrary representations might be tricky (not sure about this though, maybe this group in particular is actually easy).

[Gabri95]

Hey!

I will paste here the same answer I sent you via email for completeness

Theoretically, scale equivariance and rotation equivariance are "orthogonal" constraints which can be simultaneously enforced in the model. Currently, our library does not support scale equivariance unfortunately.

If you are interested in implementing a rotation + scale equivariant model using our library, here are a few (technical) details which might be useful to know.

Scale equivariance only enforces constraints on the radial part of the filters so one could technically implement scale equivariance by implementing a ScaleSteerableKernelBasis instead of our GaussianRadialProfile (see https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/kernels/basis.py) and then combine it with an angular steerable basis (for the rotation equivariance) through our PolarBasis class (https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/kernels/basis.py#L207).

The main implementation challenge is that one should define the direct product group "Scale X SO(2)" and use its representations to define the FieldTypes of the network's features. Its irreducible representations can be obtained by combining each irrep of the Scale group with each irrep of SO(2). However, because the Scale group is not a compact group, its representation theory is slightly different from that of compact groups such as SO(2) or O(2); this means it doesn not fit in our framework very cleanly. This is one of the reasons why we do not support scale in the library, but it doesn't mean one can't import our library and add an ad-hoc implementation of the scale group.

Apart from the implementation challenges, when building a scale + rotation equivariant model, you may find another design issue. Enforcing equivariance usually means making a number of design choices which translate into inductive biases in the network. Such choices do not only include the equivariance group to use but also the precise representations to use in each layer. As the experiments in our paper show, different choices of representations can have non-negligible impact on the performance of the model. Larger groups have a larger family of representations to choose from, making the design of equivariant networks more challenging. I believe a Scale + Rotation equivariant CNN can be a very beneficial inductive bias but it might require quite some engineering to find a good model.

P.S.: thank again @ejnnr :)