search

broncotc / bitsandbytes-rocm

MIT License
37 stars 14 forks source link

Int8 Matmul not supported on gfx1030? #1

Open gururise opened 1 year ago

gururise commented 1 year ago

Attempting to use this library on a gfx1030 (6800XT) with the huggingface transformers results in:

python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ DEBUG INFORMATION +++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable

SUCCESS!
Installation was successful!

Trying to load a simple huggingface transformer results in:

=============================================
ERROR: Your GPU does not support Int8 Matmul!
=============================================

python3: /dockerx/temp/bitsandbytes-rocm/csrc/ops.cu:347: int igemmlt(cublasLtHandle_t, int, int, int, const int8_t *, const int8_t *, void *, float *, int, int, int) [FORMATB = 3, DTYPE_OUT = 32, SCALE_ROWS = 0]: Assertion `false' failed.
Aborted (core dumped)

I am using Rocm 5.4.0 (I updated the library paths in the makefile to point to 5.4)

Jarfeh commented 1 year ago

Attempting to use this library on a gfx1030 (6800XT) with the huggingface transformers results in:

python -m bitsandbytes
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++ DEBUG INFORMATION +++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable

SUCCESS!
Installation was successful!

Trying to load a simple huggingface transformer results in:

=============================================
ERROR: Your GPU does not support Int8 Matmul!
=============================================

python3: /dockerx/temp/bitsandbytes-rocm/csrc/ops.cu:347: int igemmlt(cublasLtHandle_t, int, int, int, const int8_t *, const int8_t *, void *, float *, int, int, int) [FORMATB = 3, DTYPE_OUT = 32, SCALE_ROWS = 0]: Assertion `false' failed.
Aborted (core dumped)

I am using Rocm 5.4.0 (I updated the library paths in the makefile to point to 5.4)

May I ask what your make command was? I haven't been able to get as far as you have with my installation.

broncotc commented 1 year ago

Hi,

Currently no released RDNA card can support "Int8 mat mul", that's the original author's words for specific tensor core operations. I have explicitly deleted igemmlt as it's just not available for RDNA platform.

I have successfully used the AdamW 8 bit optimizer from this fork, but nothing else was tested.