-
Is there any plan to add support for Mixtral 8x22B?
-
Hi,
When can we expect TRT-LLM to support Smooth Quant (W8A8) quantisations for MoE models like Mixtral. Is it being planned in your roadmap? Clarity on this would be highly beneficial.
-
Hi! Thanks much for developing this tool for model merging!
It seems that the tensor names are hardcoded in https://github.com/arcee-ai/mergekit/tree/main/mergekit/_data/architectures (for Mixtral it…
-
Is it possible to do this visualization for Mixtral?
I'm interested in learning about the type of queries each of the 8 experts is good at answering.
-
### System Info
Python : 3.12.4
pandasai : 2.2.14
ibm_watsonx_ai : 0.2.6
### 🐛 Describe the bug
from pandasai import SmartDataframe
import pandas as pd
from pandasai.llm import IBMwatsonx
#…
-
### System Info
- CPU：4090 * 4
- TensorRT-LLm : v0.8.0
- CUDA Version: 12.3
- NVIDIA-SMI 545.29.06
### Who can help?
_No response_
### Information
- [X] The official example scripts
…
-
Any plan to support mixtral moe?
-
### 🐛 Describe the bug
Specifically, if we try to compile a float8 version of a FFN expert (`MixtralBlockSparseTop2MLP`), we see shape errors
Script (requires torchao and transformers):
```py…
vkuzo updated
3 weeks ago
-
Model: [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
Deployed with single card, it will report OOM error:
> (ServeController pid=207518) F…
-
Add profiling path to llm models on pytorch
- https://github.com/gclouduniverse/reproducibility/blob/main/Training/TPU-v5p/Llama2-7B-PyTorch/env.sh
- https://github.com/gclouduniverse/reproducibil…