DiNAT Checkpoints - Githubissues

SHI-Labs / Neighborhood-Attention-Transformer

Neighborhood Attention Transformer, arxiv 2022 / CVPR 2023. Dilated Neighborhood Attention Transformer, arxiv 2022

MIT License

1.05k stars 86 forks source link

DiNAT Checkpoints #56

Closed alihassanijr closed 2 years ago

alihassanijr commented 2 years ago

Checkpoints were uploaded and tested.

What's being pushed:

DiNAT, DiNAT_s, NAT-iso, DiNAT-iso, and ViT-RPB checkpoints are now available.
Markdown files are updated.
Minor typos and incorrect file names were fixed.

What's not yet pushed:

Run commands for training and validation
- They're identical to NAT so we have a temporary link to those, but will correct these later.
BFloat16 support
- Merge bf16 kernels NATTEN
- Allow amp to autocast to bf16
- Force bf16 to fall back to fp32 in torch versions < 1.12 or runs where TORCH_CUDNN_V8_API_ENABLED=0.