Bitblas supports FP8 Inference as well

neuralmagic / AutoFP8

Apache License 2.0

90 stars 13 forks source link

Bitblas supports FP8 Inference as well #2

Open LeiWang1999 opened 2 months ago

LeiWang1999 commented 2 months ago

Hello @mgoin, it's a pleasant surprise to discover this project. Thank you for your contributions to BitBLAS. We have recently added support for FP8 Matmul, hoping it will help this project.

mgoin commented 2 months ago

@LeiWang1999 Would you be able to share a minimal example for using FP8 W8A8? I would love to test this!

LeiWang1999 commented 2 months ago

checkout this dude: https://github.com/microsoft/BitBLAS/blob/main/testing/python/operators/test_general_matmul_fp8.py

LeiWang1999 commented 2 months ago

I'm recently working on quantizing an FP8 W/A language model, this project has been a great help to me, thanks :)