search

huggingface / optimum-neuron

Easy, fast and very cheap training and inference on AWS Trainium and Inferentia chips.
Apache License 2.0
177 stars 53 forks source link

Mixtral 8x7B Instruct support #394

Closed Prasanna-iq closed 1 month ago

Prasanna-iq commented 6 months ago

Hi, when can we expect the support for Mixtral 8x7B ? I can see Mistral 7B is not supported yet. Thanks

jimburtoft commented 6 months ago

@Prasanna-iq optimum-neuron supports Mistral for training. If you want to do inference on Inferentia/Trainium, check out https://huggingface.co/aws-neuron/Mistral-neuron

For Mixtral, this project needs Mixtral support in transformer-neuronx, which is almost there.

AhmedAl93 commented 3 months ago

@jimburtoft any news about this issue ? My goal is to export Mixtral 8x7b from pytorch format to neuron, while using Inf1 instance on AWS. I came to understand that optimum-neuronx/transformer-neuronx are dedicated to "Inf2, Trn1, Trn1n" instances, which aren't available in my AWS region.

HuggingFaceDocBuilderDev commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

davidshtian commented 3 months ago

As the newly launched Neuron SDK 2.18 has supported Mixtral 8x7B model (transformer_neuronx has supported this). It would be great to have this feature added in Optimum Neuron. Thanks~ 🚀

PhilipMay commented 2 months ago

Is there a timeline when the Mixtral support will be added?

dacorvo commented 2 months ago

This is already merged on main if you want to give it a try. It should be included in the next release.

HuggingFaceDocBuilderDev commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

PhilipMay commented 1 month ago

This is already merged on main if you want to give it a try. It should be included in the next release.

So this issue can be closed?

dacorvo commented 1 month ago

Yes, Mixtral is supported since 0.0.22.