Implementation of split_type_B.modReLU

[TTimTT]

Hello,

First of all, huge fan of your work! I am amazed by this library and think it will be really useful for Optical Neural Networks!

I wanted to use the following activations functions:

complextorch.nn.modules.activation.split_type_B.modReLU

But I got confused by whether or not it was already implemented so I modified it locally, like so:

class modReLU(nn.Module):

    def __init__(self, bias: float = 0.0) -> None:
        assert bias < 0, "bias must be smaller than 0 to have a non-linearity effect"

        super(modReLU, self).__init__()
        self.bias = bias

    def forward(self, z: CVTensor) -> CVTensor:

        return torch.relu(z.abs() + self.bias).mul(torch.exp(1j*z.angle()))

I was not sure what was the purpose of '_modReLU' class declared just above. Should I make a pull request instead of an issue?

Hope this small contribution may be useful.

[josiahwsmith10]

Hi TTimTT,

Thank you for your interest in this codebase. I hope you find it useful for your work!

Your implementation of modReLU will yield the correct result. I have fixed the issue in the GitHub code.

What was wrong: the modReLU class was supposed to inherit from GeneralizedPolarActivation not nn.Module (This has been fixed in the lasted commit).

Similar to the other type-B activation functions, it uses GeneralizedPolarActivation to apply the magnitude and phase activation functions, which in turn calls apply_complex_polar from complextorch.nn.functional (https://github.com/josiahwsmith10/complextorch/blob/main/complextorch/nn/functional.py).

Since a common implementation of type-B activation functions only apply a nonlinearity to the magnitude, apply_complex_polar allows phase_fun to optionally be None. In that case, it only applies mag_fun and retains the same phase. (Note: x / x.abs() is expected to outperform torch.exp(1j*x.angle) because the complex exponential is generally computationally expensive; however, they should yield identical results.)

In the init function of modReLU, the _modReLU helper function is used to define the magnitude modulus ReLU function that can be passed through to the init function of GeneralizedPolarActivation to define the magnitude and phase activation functions according to the definition of the modReLU. Although your implementation will yield the same result, I implemented it in this more generic way to reuse the apply_complex_polar functionality.

Thanks, Josiah

[TTimTT]

Hi Josiah,

I have just tested and it works like a charm! Thank you very much for your fast response time.

If I can be of any help I would be glad to help. I am just starting my PhD in optical neural net and we need a way to manipulate complex number which is why I am highly interested in your work :)

Closing the issue now as it has been resolved. Thank you again. Tim

[SantaTitular]

@TTimTT did you manage to get a network working? I can't seem to use pooling layers

[TTimTT]

Hey @SantaTitular, I was busy working on something else. I didn't try pooling layers sorry. What is the error message?