Kotlin / kotlindl

High-level Deep Learning Framework written in Kotlin and inspired by Keras
Apache License 2.0
1.46k stars 104 forks source link

Add NFResNet model to models package #57

Open zaleslaw opened 3 years ago

zaleslaw commented 3 years ago

The new family and approach of BatchNorm-free NN architectures look very perspective due to the lack of BatchNorm training support.

In the paper "High-Performance Large-Scale Image Recognition Without Normalization" the new approach is proposed.

We should apply this approach to the traditional ResNet architecture to prevent gradient exploding without BatchNorm layers.

The pre-trained weights will be a plus but are not nessesary.

zaleslaw commented 2 years ago

How to obtain pre-trained weights https://reposhub.com/python/deep-learning/benjs-nfnets_pytorch.html

zaleslaw commented 2 years ago

An example of pytorch implementation https://nfnets-pytorch.readthedocs.io/en/latest/