search

valine / NeuralFlow

Visualize the intermediate output of Mistral 7B
GNU General Public License v3.0
311 stars 13 forks source link

Mixtral 8x7b #2

Open adsharma opened 8 months ago

adsharma commented 8 months ago

Is it possible to do this visualization for Mixtral?

I'm interested in learning about the type of queries each of the 8 experts is good at answering.

valine commented 8 months ago

Definitely possible. Will need to add quantization support first as I don’t have the VRAM for half precision mixtral.