elezar
commented
1 year ago @mazoruss as a matter of interest, would CDI be an option in the long run? Although it may be feasible to add support for multiple environment variables as you suggest. Adding CDI support to docker swarm should be a longer term goal as this will ensure that future enhancements made to the NVIDIA Container Toolkit will be automatically supported.
Do you have insight into the swarm code base and whether adding CDI support is something that would be of interest?
mazoruss
Currently, allocating GPUs with docker swarm using docker services generic-resources involves modifying the /etc/nvidia-container-runtime/config.toml and adding swarm-resources = DOCKER_RESOURCE_GPU into the file. However, this assumes that there is only one type of generic resource for GPUs. In our use case, we would like to be able to allocate various different types of GPUs to different docker services, which is not possible at the moment.
For example, let’s assume there are two swarm nodes, node-A with an RTX-3080 installed and node-B with an Nvidia-T4 installed, and in the node-generic-resources on each node, they are labeled with gpu_rtx3080={uuid1} and gpu_t4={uuid2}. Now, if I were to run: $ docker service create ---generic-resource "gpu_t4=1" --name service-1 $ docker service create --generic-resource "gpu_rtx3080=1" --name service-2 I’d expect service-1 to be allocated to node-B, and service-2 to be allocated to node-A because that’s where they would have their resource demands filled. So then service-1 would be created with DOCKER_RESOUCE_GPU_T4={uuid2} in the environment variables, and service-2 would be created with DOCKER_RESOUCE_GPU_RTX3080={uuid1} in the environment variables.
However, there’s no way to define both DOCKER_RESOUCE_GPU_T4 and DOCKER_RESOUCE_GPU_RTX3080 as swarm-resource, because the config.toml file can only accept one value. So even though the docker swarm is passing in the correct environment variables, the runtime hook cannot pick up both and pass them along to NvidiaConfigs.
The proposal is to have swarm-resources accept comma delimited strings as lists. For example: swarm-resources = DOCKER_RESOURCE_GPU_RTX3080,DOCKER_RESOURCE_GPU_T4 And then the hook would pass in the correct GPU UUIDs into NvidiaConfigs based on the presence of the environment variables at runtime. This should likely be backward compatible with the existing code and would allow the flexibility for docker swarm to allocate multiple types of Nvidia GPUs intelligently.