Add support for mish activation for most cases of activation in network.

LeelaChessZero / lczero-training

For code etc relating to the network training process.

143 stars 119 forks source link

Add support for mish activation for most cases of activation in network. #193

Closed Tilps closed 2 years ago

Tilps commented 2 years ago

Requires the following added to NetworkFormat in net.proto // Activation used everywhere except head outputs or otherwise specified. enum DefaultActivation { DEFAULT_ACTIVATION_RELU = 0; DEFAULT_ACTIVATION_MISH = 1; } optional DefaultActivation default_activation = 7;