-
### System Info
CPU Architecture x86_64
GPU: 2 x NVIDIA H100
TensorRT-LLM v0.9.0
Image: tritonserver:24.05-trtllm-python-py3
Model weights: https://huggingface.co/mistralai/Mixtral-8x7B-Instruc…
-
### Checklist
- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.
- [X] 3. Please note that if the bug-related issue y…
-
Hello!
Does TensorRT-LLM supports Medusa with Mixtral 8x7B?
My understanding is that right now the Medusa [convert_checkpoint.py](https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/medusa/c…
-
## 🚀 Feature
Mixtral 8x7B is a mixture-of-experts LLM that splits the parameters in 8 distinct groups an I would like to do both training and inference with Thunder.
### Work items
- [x] Run `t…
-
config file:
```
base:
seed: &seed 42
model:
type: Mixtral
path: /models/Mixtral-8x7B-Instruct-v0.1
torch_dtype: auto
calib:
name: pileval
download: False
path: …
-
Is it possible to do this visualization for Mixtral?
I'm interested in learning about the type of queries each of the 8 experts is good at answering.
-
### System Info
```shell
Name: optimum
Version: 1.18.0.dev0
Name: transformers
Version: 4.36.0
Name: auto-gptq
Version: 0.6.0.dev0+cu118
CUDA Version: 11.8
Python 3.8.17
```
### Who can help…
-
### System Info
text-generation-inference version 2.2.0
model "mistralai/Mixtral-8x7B-Instruct-v0.1"
### Information
- [X] Docker
- [ ] The CLI directly
### Tasks
- [X] An officially supported c…
-
Add reference code for `mixtral-8x7b`(https://github.com/mlcommons/inference/tree/master/language/mixtral-8x7b) in `axs`.
To do the following steps:
- add recipe for downloading dataset
- add recip…
-
Can you confirm if Mixtral is currently supported, e.g., `mistralai/Mixtral-8x7B-Instruct-v0.1?` I saw in another issue that Mistral is supported, but I'm not sure about Mixtral-8x7B since it's a diff…