Baseline & Long Range Arena

[kmheckel]

General approach:

• FP training as a reference baseline / target

• PTQ as a baseline for quantization

• QAT results

• (optional) QAT starting from the FP checkpoint (i.e., finetuning)

  • analysis of compute (during training) vs. performance (accuracy) vs. efficiency (# of bits / compression)

• [x] Sequential MNIST @stevenabreu7

  • [x] PTQ results @stevenabreu7

LRA:

• [x] sequential CIFAR @stevenabreu7
  • [x] PTQ results @stevenabreu7
• [x] ListOps @stevenabreu7
  • [x] PTQ results @stevenabreu7
  • [ ] improve QAT results (all are bad..)
• [x] Text (imdb) @stevenabreu7
  • [x] PTQ results @stevenabreu7
  • [ ] improve QAT results (only W4 and W2 with W_a8 worked..)
• [ ] Retrieval @AlessandroPierro
  • [ ] PTQ results
• [ ] Pathfinder @AlessandroPierro
  • [ ] PTQ results
• [ ] Path-X @AlessandroPierro
  • [ ] PTQ results

Bonus:

• [ ] Speech Commands 35

[stevenabreu7]

Isn't LRA with sequential CIFAR-10 instead of MNIST?

[kmheckel]

You're probably right - I didn't copy down all of the benchmarks and was basing the list off what was in the S5 repo so this may not be a perfect list. Will fix it.

On Wed, May 1, 2024, 22:45 Steven Abreu @.***> wrote:

Isn't LRA with sequential CIFAR-10 instead of MNIST?

— Reply to this email directly, view it on GitHub https://github.com/kmheckel/NeuroSSMs/issues/9#issuecomment-2089186568, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMG7YKBFQEIRUXT527CFPMTZAFPBDAVCNFSM6AAAAABHCGBJQWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBZGE4DMNJWHA . You are receiving this because you authored the thread.Message ID: @.***>

[stevenabreu7]

We could also just do both (since MNIST is relatively fast)

[AlessandroPierro]

Retrieval

• FP16: in progress
• W8A8: in progress
• W4A8: in progress
• W2A8: in progress

Pathfinder

• FP16: DONE
• W8A8: in progress
• W4A8: in progress
• W2A8: in progress

Path-X