This is an implementation of the cyclemoid activation function for PyTorch.

The cyclemoid function achieved state-of-the-art results in a recent benchmark with other popular activation functions as shown below:

Note that this is a figure from the paper submitted on April 1st, 2022. An arxiv preprint will be uploaded soon.

Installation

You can install the cyclemoid-pytorch package via

pip install cyclemoid_pytorch

Usage

This package implements a CycleMoid class and a cyclemoid function. You can use these are drop-in replacements for any activation in PyTorch. For example

from cyclemoid_pytorch import CycleMoid

torch.nn.Sequential(
    # ...,
    CycleMoid(), # instead of torch.nn.ReLU()
    # ...
    )

or

from cyclemoid_pytorch import cyclemoid

# ...
def forward(self, x):
    # ...
    x = cyclemoid(x) # instead of torch.sigmoid(x)

Visualization

import matplotlib.pyplot as plt
import torch
from cyclemoid_pytorch import cyclemoid

x = torch.arange(-5, 5, 0.01)
y = cyclemoid(x)
plt.plot(x, y)

Demo

For a concrete usage, check out the demo notebook.

Appendix

You can now also use the cyclemoid activation in Keras.

import tensorflow as tf
from cyclemoid_pytorch.easteregg import CycleMoid

tf.keras.utils.get_custom_objects()['cyclemoid'] = CycleMoid

model = tf.keras.Sequential(
    [
        tf.keras.Input(...),
        tf.keras.layers.Conv2D(..., activation="cyclemoid"),
        # ... 
    ]
)