ModReLU : bug?

[pfeatherstone]

Currently the implementation is:

def modrelu(input, threshold=0.5):
    r"""Compute the modulus relu of the complex tensor in re-im pair."""
    # scale = (1 - \trfac{b}{|z|})_+
    modulus = torch.clamp(abs(input), min=1e-5)
    return input * torch.relu(1. - threshold / modulus)

Shouldn't it be:

def modrelu(input, threshold=0.5):
    r"""Compute the modulus relu of the complex tensor in re-im pair."""
    # scale = (1 - \trfac{b}{|z|})_+
    modulus = torch.clamp(abs(input), min=1e-5)
    return input * torch.relu(1. + threshold / modulus)

i.e. torch.relu(1. - threshold / modulus) -> torch.relu(1. + threshold / modulus)

?

The paper states:

g(z) = ReLU (|z|+ b) exp {iφ(z)} 

not

g(z) = ReLU (|z|- b) exp {iφ(z)} 

[pfeatherstone]

Using native pytorch complex tensors, the activation should be:

z = torch.relu(torch.abs(z) + b) * torch.exp(1.j * torch.angle(z)) 

which can be re-written as:

z = z * torch.relu(1.0 + b / torch.abs(z))

[pfeatherstone]

so yeah, pretty sure that's a bug

[ivannz]

This whole mod-relu activation looks very much much like soft-thresholding operator from ell-1 regularization. And I recall deciding that flipped sign parameterization would make more sense, since intuitively b specifies the clipping threshold.

As such, this is not a bug, since the sign of b is just flipped. So to clip the modulus at 0.5 you need to call modrelu(z, 0.5) (current implementation) instead of modrelu(z, -0.5) (as in the proposed bugfix). To me the second variant seemed back then and seems now highly counterintuitive. I guess i should've indicated this in the docstring, since it deviates form the original paper.

[pfeatherstone]

Maybe just some updated documentation would be sufficient then.

[pfeatherstone]

Great docs! Cheers