Cannot reproduce the result of ConvNeXt pretraining

[iShohei220]

🐛 Describe the bug

I tried to reproduce the result of ConvNeXt-Tiny's pretraining reported here using the official training recipe. However, the result was worse than the reported score (i.e., 82.52 % top-1 acc and 96.146 % top-5 acc). My result was 81.426 % at top-1 and 95.338 % at top-5. Is it just due to the choice of random seed? If so, please share it.

I also wonder why this training recipe requires so long time (i.e., 600 epochs) for training, while the official implementation of ConvNext can be trained in 300 epochs.

Versions

PyTorch version: 2.1.0a0+fe05266 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0 Clang version: Could not collect CMake version: version 3.24.1 Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0] (64-bit runtime) Python platform: Linux-4.18.0-193.el8.x86_64-x86_64-with-glibc2.29 Is CUDA available: True CUDA runtime version: 12.1.66 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A100-SXM4-40GB Nvidia driver version: 525.105.17 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0 /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 52 bits physical, 57 bits virtual CPU(s): 144 On-line CPU(s) list: 0-143 Thread(s) per core: 2 Core(s) per socket: 36 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 106 Model name: Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz Stepping: 6 Frequency boost: enabled CPU MHz: 3185.813 CPU max MHz: 2401.0000 CPU min MHz: 800.0000 BogoMIPS: 4800.00 Virtualization: VT-x L1d cache: 3.4 MiB L1i cache: 2.3 MiB L2 cache: 90 MiB L3 cache: 108 MiB NUMA node0 CPU(s): 0-35,72-107 NUMA node1 CPU(s): 36-71,108-143 Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid md_clear pconfig flush_l1d arch_capabilities

Versions of relevant libraries: [pip3] numpy==1.22.2 [pip3] pytorch-fid==0.2.1 [pip3] pytorch-quantization==2.1.2 [pip3] torch==2.1.0a0+fe05266 [pip3] torch-tensorrt==1.4.0.dev0 [pip3] torchtext==0.13.0a0+fae8e8c [pip3] torchvision==0.15.0a0 [pip3] triton==2.0.0 [conda] Could not collect

[NicolasHug]

We don't set a seed when training, what we usually do is run a bunch of training jobs (with uncontrolled seeds) and we pick the median. It's possible that you got a bit unlucky with your run?

You'll find more details about the training in the original PR https://github.com/pytorch/vision/pull/5197

[iShohei220]

Thank you for your quick reply.

We don't set a seed when training, what we usually do is run a bunch of training jobs (with uncontrolled seeds) and we pick the median. It's possible that you got a bit unlucky with your run?

Okay, I will run some more trials, and report the results.

You'll find more details about the training in the original PR https://github.com/pytorch/vision/pull/5197

Thank you for your reference. I read the PR, but I am still not sure about why the training takes so long compared to ConvNeXt's official implementation. The committer (@datumbox) mentioned that he followed Torchvision's new recipe, but it seems that the new recipe is not used for the other models (e.g., ResNet) from what I have read in the README. Is there any special reason why only ConvNeXt requires longer training than the others?

[NicolasHug]

We [re]trained a lot of the models with the goal to push the accuracy further, and increasing the number of epochs was part of it. It's not super obvious but a lot of the older models also use more epochs, it's not just ConvNeXt. The V2 version of resnets also used 600. The way we document the training recipes is a bit scattered, the easiest way is probably to look at the weight table in the docs https://pytorch.org/vision/main/models.html#table-of-all-available-classification-weights and click on the corresponding "recipe" link. For Resnet50 V2 you'll end up here

[iShohei220]

Thanks! I understand the background. I hope the README will be updated to the new recipe for all models, since the current documentation is a little confusing.

[NicolasHug]

That's fair @iShohei220 . Maybe we can add a very visible header at the top of those README to indicate the best way to look at the up-to-date recipes. Would you like to submit a PR for that?

[iShohei220]

Would you like to submit a PR for that?

Sure! I will submit it within a week.

[iShohei220]

Additional report

I've run 3 training jobs, but the results were still worse than the reported accuracy. See the W&B report for more details.

It seems that the models consistently start to overfit in the final 150 epochs. The best accuracy at around the 450th epoch is close to the reported value. If you picked it up as the pretrained model, it makes sense, but it sounds a little tricky.

I think it would be better to strengthen regularization (e.g., increase the weight decay) to prevent the over-fitting.

[iShohei220]

I hope someone else will reproduce the result and confirm that this phenomenon (i.e., the accuracy degradation and over-fitting) is not specific to my environmet.