search

bottlerocket-os / bottlerocket

An operating system designed for hosting containers
https://bottlerocket.dev
Other
8.78k stars 519 forks source link

[EKS] Support Inferentia/Neuron Runtime #1995

Open samjo-nyang opened 2 years ago

samjo-nyang commented 2 years ago

What I'd like: I think it requires the neuron driver on https://github.com/aws/aws-neuron-sdk

Any alternatives you've considered: Nothing

cbgbt commented 2 years ago

Thanks for raising this. We're interested in integrating with Neuron, and it's something we're planning to look into down the road!

cbgbt commented 2 years ago

Re-titled this to be consistent with #1075, which is similar but for an ECS Inferentia variant.

stmcginnis commented 1 year ago

Is this still needed?

samjo-nyang commented 1 year ago

Yes, we are using more neuron instances than I created the ticket. (actively migrating workloads from gpu to neuron)

hustshawn commented 1 year ago

Container SSA check-in. IHAC is running ML workloads with Inferentia on EKS. They are quite interested in Bottlerocket in terms of awesome security benefits they get with less overhead. They really want to align the company standards to use Bottlerocket for general business application as well as ML workloads. But the lack of support for Inferentia would affect their adoption.

heichow commented 1 year ago

IHAC who is running Stable Diffusion on EKS Inf2, and they wish to adopt Bottlerocket image cache solution to reduce the large image (10+GB) pulling time from ECR around 3-4 minutes. Foreseeing the increasing GenAI model hosting with Inferentia, supporting Inferentia/Neuron runtime will have a big impact.