block scaling support not found

[complexfilter]

The FA3 paper says:

Accuracy: block quantization and incoherent processing. With FP8 (e4m3) format, one only uses 3 bits to store the mantissa and 4 bits for the exponent. This results in higher numerical error than FP16/BF16. Moreover, large models typically have outlier values [20, 54] that are much larger in magnitude than most other values, making quantization difficult. One typically use per-tensor scaling [37] by keeping one scalar per tensor (e.g., one for Q, for K, and for V). To reduce the numerical error of attention in FP8, we employ two techniques: 1. Block quantization: we keep one scalar per block, so that for each of Q, K, V we split the tensor into blocks of size 𝐵𝑟 × 𝑑 or 𝐵𝑐 × 𝑑 and quantize them separately. This quantization can be fused with an operation right before attention (e.g., rotary embedding) with no additional slow down (since rotary embedding is memory-bandwidth bound). As the FlashAttention-3 algorithm naturally operates on blocks, we can scale each block of S to account for this block quantization at no computation cost.

but i don’t see any support for block scaling in the actual repo.

[cly12188]

The FA3 paper says:

Accuracy: block quantization and incoherent processing. With FP8 (e4m3) format, one only uses 3 bits to store the mantissa and 4 bits for the exponent. This results in higher numerical error than FP16/BF16. Moreover, large models typically have outlier values [20, 54] that are much larger in magnitude than most other values, making quantization difficult. One typically use per-tensor scaling [37] by keeping one scalar per tensor (e.g., one for Q, for K, and for V). To reduce the numerical error of attention in FP8, we employ two techniques: 1. Block quantization: we keep one scalar per block, so that for each of Q, K, V we split the tensor into blocks of size 𝐵𝑟 × 𝑑 or 𝐵𝑐 × 𝑑 and quantize them separately. This quantization can be fused with an operation right before attention (e.g., rotary embedding) with no additional slow down (since rotary embedding is memory-bandwidth bound). As the FlashAttention-3 algorithm naturally operates on blocks, we can scale each block of S to account for this block quantization at no computation cost.

but i don’t see any support for block scaling in the actual repo. Block quantization seems to have its code publicly available, but it appears that the code for Incoherent processing has not been released. Have you found the implementation location for Incoherent processing?

[goldhuang]

The FA3 paper says:

Accuracy: block quantization and incoherent processing. With FP8 (e4m3) format, one only uses 3 bits to store the mantissa and 4 bits for the exponent. This results in higher numerical error than FP16/BF16. Moreover, large models typically have outlier values [20, 54] that are much larger in magnitude than most other values, making quantization difficult. One typically use per-tensor scaling [37] by keeping one scalar per tensor (e.g., one for Q, for K, and for V). To reduce the numerical error of attention in FP8, we employ two techniques: 1. Block quantization: we keep one scalar per block, so that for each of Q, K, V we split the tensor into blocks of size 𝐵𝑟 × 𝑑 or 𝐵𝑐 × 𝑑 and quantize them separately. This quantization can be fused with an operation right before attention (e.g., rotary embedding) with no additional slow down (since rotary embedding is memory-bandwidth bound). As the FlashAttention-3 algorithm naturally operates on blocks, we can scale each block of S to account for this block quantization at no computation cost.

but i don’t see any support for block scaling in the actual repo. Block quantization seems to have its code publicly available, but it appears that the code for Incoherent processing has not been released. Have you found the implementation location for Incoherent processing?

@cly12188 Could you share the link of block quantization implementation?