LangStream / langstream

LangStream. Event-Driven Developer Platform for Building and Running LLM AI Apps. Powered by Kubernetes and Kafka.
https://langstream.ai
Apache License 2.0
386 stars 28 forks source link

Support GPU in Kubernetes cluster #758

Open ddieruf opened 8 months ago

ddieruf commented 8 months ago

I would like to use the gpu in my K8s cluster with my LS application. Could a new feature be added to the resources area in the pipeline manifest :

...
resources:
  gpu:
    vendor: "gpu-vendor.example/example-gpu"
    count: 1

Which would be incorporated into the agent statefulset template, to include the limits spec:

...
resources:
  limits:
    gpu-vendor.example/example-gpu: 1

Then in my custom python processor, I can do image manipulations and bring in my own models to do fun things :)