search

kurtosis-tech / kurtosis

A platform for packaging and launching ephemeral backend stacks with a focus on approachability for the average developer.
https://docs.kurtosistech.com/
Apache License 2.0
380 stars 53 forks source link

Add Docker GPU support #1898

Open igor-ritual opened 11 months ago

igor-ritual commented 11 months ago

Background & motivation

Docker requires special deployment resources to enable GPU support:

https://docs.docker.com/compose/gpu-support/

Desired behaviour

For Docker, use a new ServiceConfig argument to set the deploy resources.

How important is this to you?

Critical; Kurtosis is unusable for me without it.

What area of the product does this pertain to?

SDK: the Software Development Kit libraries - Typescript, Go, etc.

leeederek commented 11 months ago

Hey @igor-ritual - thanks for filing this! Can you share with me some more details on your workflow (i.e. why you need GPU support)? This information will help us better understand how Kurtosis is being used/examined in the wild and will help us prioritize FRs like this :) Thanks!

igor-ritual commented 11 months ago

Hi @leeederek , I would like to run TGI under kurtosis and the docs show that it needs special cli arguments when run under docker:

docker run --gpus all --shm-size 1g

https://github.com/huggingface/text-generation-inference

leeederek commented 11 months ago

Gotcha - thats useful. I'm acknowledging receipt of this request now and will circle back to you with a decision! One more question: are you running this locally or on a remote server and for how long?

igor-ritual commented 11 months ago

I am running this locally on a Mac for development and remotely on an ubuntu server permanently.