[QUESTION] What is the difference between optimum-neuron and transformers-neuronx? - Githubissues

huggingface / optimum-neuron

Easy, fast and very cheap training and inference on AWS Trainium and Inferentia chips.

Apache License 2.0

177 stars 53 forks source link

[QUESTION] What is the difference between optimum-neuron and transformers-neuronx? #460

Closed leoribeiro closed 3 months ago

leoribeiro commented 5 months ago

I would like to understand the differences between this optimum-neuron and transformers-neuronx.

dacorvo commented 5 months ago

Basically optimum-neuron uses parts of transformers-neuronx.

transformers-neuronx:

demonstrates the ability to run CausalLM models on Neuron hardware using pre-fetched Hugging Face models,
custom generation API,
no TGI support,
maintained by AWS,
not open to contributions.

optimum-neuron:

allows to train and run many models from the Hugging Face hub (not only LLM) on Neuron Hardware,
fully integrated with the Hugging Face hub (including serialization and caching),
transformers generation API,
TGI support,
maintained by Hugging Face,
open to contributions.