[QUESTION] Unable to use ReLU as activation function

[nkw011]

I am currently training my custom metric using the UnifiedMetric and want to use ReLU instead of the default Tanh. I thought simply replacing "Tanh" with "ReLU" in yaml file might work. However, I am encountering the following error message: Exception: ReLU is not a valid activation function! I want to know if there is a way to use ReLU as the activation function.

Additionally, It seems that an error occurs at activation.title().

def build_activation(self, activation: str) -> nn.Module:
    if hasattr(nn, activation.title()):
        return getattr(nn, activation.title())()
    else:
        raise Exception(f"{activation} is not a valid activation function!")

Thankyou‼️

[ricardorei]

Hey @nkw011! The build_activation function seems to be lacking some flexibility.

I think the best thing to do is to just modify it to be the following:

def build_activation(self, activation: str) -> nn.Module:
    if hasattr(nn, activation):
        return getattr(nn, activation)()
    else:
        raise Exception(f"{activation} is not a valid activation function!")

and then everything should work.

[ricardorei]

the reason its not working atm is that when you pass "ReLU" it will be converted to the string "Relu" by the title function and thus it will not match any of the activation functions in the torch.nn package

[nkw011]

@ricardorei Thank you for replying my question! I'm sorry but have one more question. The build activation function is located in comet/modules/feedforward.py. Can I get a way to modify the bulid activation function in pip package 'unbabel-comet' installed in local computer as you explain?

[ricardorei]

You should be able to do it. When you installed the package the code was stored somewhere in your computer. You just have to locate the source code and modify the function.

A simpler alternative is to just install the code from source:

Just clone the repo to your favourite location and install it:

git clone https://github.com/Unbabel/COMET
cd COMET
pip install poetry
poetry install

Then you can modify whatever you like.