Can I use a CUDA kernel with a model quantized using triton & vice-versa?

[vedantroy]

I'm assuming that it is OK to:

• Quantize a model using Triton, and then run inference using CUDA
• The opposite of the above

Is this true?

[fpgaminer]

If by CUDA you mean GPTQ-for-LLaMa or similar, then the algorithm is the same, and the raw data is the same, but the format the quantized model is stored in might be slightly different.

[vedantroy]

If by CUDA you mean GPTQ-for-LLaMa or similar, then the algorithm is the same, and the raw data is the same, but the format the quantized model is stored in might be slightly different.

I'm assuming that if I store everything in the .safetensors format, it should be fine / interchangable? Maybe this is not the case?

[fpgaminer]

I meant more that, for example, my quantize script stores values in float16 since that's what the Triton kernels expect, whereas historically GPTQ-for-LLaMa stored and used them as float32.