ModelZoo for Pytorch

This is a model zoo project under Pytorch. In this repo I will implement some of basic classification models which have good performance on ImageNet. Then I will train them in most fair way as possible and try my best to get SOTA model on ImageNet. In this repo I'll only consider FP16.

Usage

Environment

• OS: Ubuntu 18.04
• CUDA: 10.1, CuDNN: 7.6
• Devices: I use 8 RTX 2080ti(8 V100 should be much better /cry). This project is in FP16 precision, it's recommend to use FP16 friendly devices like RTX series, V100. If you want to totally reproduce my research, you'd better use same batch size with me.

Requirement

• Pytorch: >= 1.6.0 (Need torch.cuda.amp in version 1.6)
• TorchToolbox: stable version. Helper functions to make your code simpler and more readable, it's a optional tools if you don't want to use it just write them yourself.

LMDB Dataset

• No necessary.

If you found any IO bottleneck please use LMDB format dataset. A good way is try both and find out which is more faster.

I provide conversion script here.

Train script

python distribute_train_script --params

Here is a example

python distribute_train_script.py --data-path /s4/piston/ImageNet --batch-size 256 --dtype float16 \
                                  -j 48 --epochs 360 --lr 2.6 --warmup-epochs 5 --label-smoothing \
                                  --no-wd --wd 0.00003 --model GhostNet --log-interval 150 --model-info \
                                  --dist-url tcp://127.0.0.1:26548 --world-size 1 --rank 0

ToDo

• [x] Resume training
• Try Nvidia-DALI
• [x] Multi-node(distributed) training by Apex or BytePS Pytorch
• [x] I may try AutoAugment.This project aims to train models by ourselves to observe and learn, it's impossible for me to train this, just copy feels meaningless.

Baseline models

• I use nesterov SGD and cosine lr decay with 5 warmup epochs by default[2][3] (to save time), it's more common and effective.
• Batch size is pre GPU holds. Total batch size should be (batch size gpus).

Optimized Models(with tricks)

• In progress.

Ablation Study on Tricks

Here are lots of tricks to improve accuracy during this years.(If you have another idea please open an issue.) I want to verify them in a fair way.

Tricks: RandomRotation, OctConv[14], Drop out, Label Smoothing[4], Sync BN, SwitchNorm[6], Mixup[17], no decay bias[7], Cutout[5], Relu6[18], swish activation[10], Stochastic Depth[9], Lookahead Optimizer[11], Pre-active(ResnetV2)[12], DCNv2[13], LIP[16].

• Delete line means make me out of memory.

Special: Zero-initialize the last BN, just call it 'Zero γ', only for post-active model.

I'll only use 120 epochs and 128*8 batch size to train them. I know some tricks may need train more time or larger batch size but it's not fair for others. You can think of it as a performance in the current situation.

• More epochs for Mixup, Cutout, Dropout may get better results.
• Auto/Rand Augment may train 180 epochs better.

Citation

@misc{ModelZoo.pytorch,
  title = {Basic deep conv neural network reproduce and explore},
  author = {X.Yang},
  URL = {https://github.com/PistonY/ModelZoo.pytorch},
  year = {2019}
  }

Reference

• [1] Deep Residual Learning for Image Recognition
• [2] Accurate, Large Minibatch SGD:Training ImageNet in 1 Hour
• [3] Bag of Tricks for Image Classification with Convolutional Neural Networks
• [4] Rethinking the Inception Architecture for Computer Vision
• [5] Improved Regularization of Convolutional Neural Networks with Cutout
• [6] Differentiable Learning-to-Normalize via Switchable Normalization OpenSourse
• [7] Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• [8] MIXED PRECISION TRAINING
• [9] Deep Networks with Stochastic Depth
• [10] SEARCHING FOR ACTIVATION FUNCTIONS
• [11] Lookahead Optimizer: k steps forward, 1 step back
• [12] Identity Mappings in Deep Residual Networks
• [13] Deformable ConvNets v2: More Deformable, Better Results
• [14] Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• [15] Training Neural Nets on Larger Batches: Practical Tips for 1-GPU, Multi-GPU & Distributed setups
• [16] LIP: Local Importance-based Pooling
• [17] mixup: BEYOND EMPIRICAL RISK MINIMIZATION
• [18] Gradient Centralization: A New Optimization Technique for Deep Neural Networks