[Feature]: Build and publish Neuron docker image

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

https://docs.vllm.ai

Apache License 2.0

30.7k stars 4.66k forks source link

[Feature]: Build and publish Neuron docker image #4838

Open yaronr opened 6 months ago

yaronr commented 6 months ago

🚀 The feature, motivation and pitch

It seems like the current docker images don't support Neuron (Inferentia). It would be very helpful if there was a tested, managed Neuron docker image to use. While at the same subject, it would be even better if some documentation would be added on running vLlm Neuron using containers.

Alternatives

DJL?

Additional context

No response

github-actions[bot] commented 4 weeks ago

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

dtrifiro commented 3 weeks ago

I guess this can be closed? https://github.com/vllm-project/vllm/tree/main/Dockerfile.neuron

yaronr commented 3 weeks ago

Hi @dtrifiro To close this, there would need to be an official docker image (not docker file) here, or on docker hub.