-
**Describe the bug**
After a model is generated running `big_model_fp8.py`, lm_eval dont not work unless the .py files from the original base model is transferred to the generated model folder. Happe…
-
### Your question
I got:
Total VRAM 8188 MB, total RAM 16011 MB
pytorch version: 2.3.1+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4060 Laptop GPU : cudaMallocAsync
…
-
Hi, I am trying to use fp8 with TransformerEngine. I am using a version of GPT-Neox repo, which uses deepspeed.
I can get fp8 to run in my MLPs with model parallel, but when I use pipeline paralle…
-
Is there a way to run these models with 12 GB RAM?
With fp8 models it is working but with GGUF models it always fail.
-
`Flux diffusion model implementation using quantized fp8 matmul & remaining layers use faster half precision accumulate, which is ~2x faster on consumer devices.`
Hello there!
Thanks for sharing you…
-
I use a1111 v1.8.0 with fp8 weight option enabled, and an error occurred.
"RuntimeError: mat1 and mat2 must have the same dtype, but got Half and Float8_e4m3fn"
Without fp8 weight option, the erro…
-
[context_flashattention_nopad_fp16_fp8.txt](https://github.com/user-attachments/files/16421521/context_flashattention_nopad_fp16_fp8.txt)
we have implemented a f8 version of context_flashattention_…
-
Hi, how to cast a float/bfloat16 tensor to fp8? I want to conduct W8A8 (fp8) quantization. But I didn't find an example of quantizing act to FP8 format.
-
There is a very large performance regression (6x slower for `[8192,8192]x[8192,8192]`) when using Triton for matmuls with float8 e4m3 inputs, comparing `2.2.0` and `2.3.0`.
We use Triton for our [f…
mgoin updated
3 months ago
-
## 🚀 Feature
CuDNN provides flexible support for performant gemm/conv with fp8 quantization. Thunder introducing fp8 casts in its traces can benefit from cudnn fusions.
### Motivation
Today, thu…