Features

• Single GPU sharing among multiple processes/containers
• Memory and fault isolation is guaranteed because co-located processes use different CUDA contexts, unlike other approaches such as NVIDIA MPS.
• Completely transparent to applications, no code changes needed
• Each process/container has whole GPU memory available
  • Uses Unified Memory to swap GPU memory to system RAM
  • Scheduler optionally serializes overlapping GPU work to avoid thrashing (assigns exclusive access to one app for TQ seconds at a time)
  • Apps release GPU if done with work before TQ elapses
• Device plugin for Kubernetes

Key Idea

1. With cudaMalloc(), the sum of memory allocations from CUDA apps must be smaller than physical GPU memory size (Σ(mem_allocs) <= GPU_mem_size).
2. Hooking and replacing all cudaMalloc() calls in an application with cudaMallocManaged(), i.e., transparently forcing the use of CUDA's Unified Memory API does not affect correctness and only leads to a ~1% slowdown.
3. If we apply (2), constraint (1) no longer holds for an application written using cudaMalloc().
4. When we oversubscribe GPU memory (Σ(mem_allocs) > GPU_mem_size), we must take care to avoid thrashing when the working sets of co-located apps (i.e., the data they are actively using) don't fit in GPU mem (Σ(wss) > GPU_mem_size). We use nvshare-scheduler to serialize work on the GPU to avoid thrashing. If we don't serialize the work, the frequent (every few ms) context switches of NVIDIA's black-box scheduler between the co-located apps will cause thrashing.
5. If we know that Σ(wss) <= GPU_mem_size, we can disable nvshare-scheduler's anti-thrashing mode.

Supported GPUs

nvshare relies on Unified Memory's dynamic page fault handling mechanism introduced in the Pascal microarchitecture.

It supports any Pascal (2016) or newer Nvidia GPU.

It has only been tested on Linux systems.

Overview

nvshare components

• nvshare-scheduler, which is responsible for managing a single Nvidia GPU. It schedules the GPU "lock" among co-located clients that want to submit work on the GPU. It assigns exclusive access to the GPU to clients in an FCFS manner, for TQ seconds at a time.
• libnvshare.so, which we inject into CUDA applications through LD_PRELOAD and which:
  • Interposes (hooks) the application's calls to the CUDA API, converting normal memory allocation calls to their Unified Memory counterparts
  • Implements the client side of nvshare, which communicates with the nvshare-scheduler instance to gain exclusive access to the GPU each time the application wants to do computations on the GPU.
• nvsharectl, which is a command-line tool used to configure the status of an nvshare-scheduler instance.

Some Details on nvshare-scheduler

IMPORTANT: nvshare currently supports only one GPU per node, as nvshare-scheduler is hardcoded to use the Nvidia GPU with ID 0.

nvshare-scheduler's job is to prevent thrashing. It assigns exclusive usage of the whole GPU and its physical memory to a single application at a time, handling requests from applications in an FCFS manner. Each app uses the GPU for at most TQ seconds. If the app is idle, it releases the GPU early. When it wants to compute something on the GPU at a later point, it again requests GPU access from the scheduler. When the scheduler gives it access to the GPU, the app gradually fetches its data to the GPU via page faults.

If the combined GPU memory usage of the co-located applications fits in the available GPU memory, they can seamlessly run in parallel.

However, when the combined memory usage exceeds the total GPU memory, nvshare-scheduler must serialize GPU work from different processes in order to avoid thrashing.

The anti-thrashing mode of nvshare-scheduler is enabled by default. You can configure this using nvsharectl. We currently have no way of automatically detecting thrashing, therefore we must toggle the scheduler on/off manually.

Memory Oversubscription For a Single Process

nvshare allows each co-located process to use the whole physical GPU memory. By default, it doesn't allow a single process to allocate more memory than the GPU can hold, as this can lead to internal thrashing for the process, regardless of the existence of other processes on the same GPU.

If you get a CUDA_ERROR_OUT_OF_MEMORY it means that your application tried to allocate more memory than the total capacity of the GPU.

You can set the NVSHARE_ENABLE_SINGLE_OVERSUB=1 environment variable to enable a single process to use more memory than is physically available on the GPU. This can lead to degraded performance.

The Scheduler's Time Quantum (TQ)

The TQ has effect only when the scheduler's anti-thrashing mode is enabled.

A larger time quantum sacrifices interactivity (responsiveness) in favor of throughput (utilization).

The scheduler's TQ dictates the amount of time the scheduler assigns the GPU to a client for. A larger time quantum sacrifices interactivity (latency) in favor of throughput (utilization) and vice-versa.

You shouldn't set the time quantum to a very small value (< 10), as the time spent fetching the pages of the app that just acquired the GPU lock takes a few seconds, so it won't have enough time to do actual computations.

To minimize the overall completion time of a set of sequential (batch) jobs, you can set the TQ to very large value.

Without nvshare, you would run out of memory and have to run one job after another.

With nvshare:

• Only the GPU portions of the jobs will run serialized on the GPU, the CPU parts will run in parallel
• Each application will hold the GPU only while it runs code on it (due to the early release mechanism)

Further Reading

Deploy on a Local System

Installation (Local)

Usage (Local)

Test (Local)

If you don't want to use docker, you can run the tests manually by cloning the repo, going to the tests/ directory and running the Python programs by hand, using LD_PRELOAD=libnvshare.so. The default tests below use about 10 GB GPU memory each. Use these if your GPU has at least 10 GB memory.

Deploy on Kubernetes

Installation (Kubernetes)

Requirements:

Usage (Kubernetes)

Use an nvshare.com/gpu Device in Your Container

In order to use an nvshare virtual GPU, you need to request an 'nvshare.com/gpu' device in the limits section of the resources of your container.

(Optional) Configure an nvshare-scheduler instance using nvsharectl

As the scheduler is a DaemonSet, there is one instance of nvshare-scheduler per node.

1. Store the Pod name of the instance you want to change in a variable:

   You can use kubectl get pods -n nvshare-system to find the name.

     NVSHARE_SCHEDULER_POD_NAME=<pod-name>

2. Execute into the container and use nvsharectl to reconfigure the scheduler:

     kubectl exec -ti ${NVSHARE_SCHEDULER_POD_NAME?} -n nvshare-system -- nvsharectl ...

Test (Kubernetes)

1. Deploy the test workloads:

   The default tests below use about 10 GB GPU memory each. Use these if your GPU has at least 10 GB memory. Alternatively, you can pick any in the tests/manifests directory. The *-small variants use less GPU memory. You can either clone the repo or copy the link to the raw file and pass it to kubectl.

     kubectl apply -f https://raw.githubusercontent.com/grgalex/nvshare/main/tests/kubernetes/manifests/nvshare-tf-pod-1.yaml && \
     kubectl apply -f https://raw.githubusercontent.com/grgalex/nvshare/main/tests/kubernetes/manifests/nvshare-tf-pod-2.yaml

2. In a terminal window, watch the logs of the first Pod:

     kubectl logs nvshare-tf-matmul-1 -f

3. In another window, watch the logs of the second Pod:

     kubectl logs nvshare-tf-matmul-2 -f

4. (Optional) Find the node that the Pods are running on, watch the nvshare-scheduler logs from that node

5. Delete the test workloads:

     kubectl delete -f https://raw.githubusercontent.com/grgalex/nvshare/main/tests/kubernetes/manifests/nvshare-tf-pod-1.yaml && \
     kubectl delete -f https://raw.githubusercontent.com/grgalex/nvshare/main/tests/kubernetes/manifests/nvshare-tf-pod-2.yaml

Uninstall (Kubernetes)

Delete all nvshare components from your cluster:

kubectl delete -f https://raw.githubusercontent.com/grgalex/nvshare/main/kubernetes/manifests/scheduler.yaml
kubectl delete -f https://raw.githubusercontent.com/grgalex/nvshare/main/kubernetes/manifests/device-plugin.yaml && \
kubectl delete -f https://raw.githubusercontent.com/grgalex/nvshare/main/kubernetes/manifests/nvshare-system-quotas.yaml && \
kubectl delete -f https://raw.githubusercontent.com/grgalex/nvshare/main/kubernetes/manifests/nvshare-system.yaml && \

Build For Local Use

These instructions assume building on a Debian-based system.

You can use the artifacts on any machine that has glibc and supports the ELF binary format.

Build Docker Images

Future Improvements

• nvshare currently supports only one GPU per node, as the nvshare-scheduler is hardcoded to use the Nvidia GPU with ID 0. Support multiple GPUs per node/machine.
• Automatically detect thrashing, optimally toggle the nvshare-scheduler on/off.
• Intra-node GPU migration.
• Inter-node GPU migration.

Feedback

• Open a Github issue on this repository for any questions/bugs/suggestions.
• If your organization is using nvshare, you can drop me a message/mail and I can add you to USERS.md.

Cite this work

If you found this work useful, you can cite it in the following way:

Georgios Alexopoulos and Dimitris Mitropoulos. 2024. nvshare: Practical
GPU Sharing without Memory Size Constraints. In 2024 IEEE/ACM 46th
International Conference on Software Engineering: Companion Proceedings
(ICSE-Companion ’24), April 14–20, 2024,Lisbon, Portugal. ACM, New York,
NY, USA, 5 pages. https://doi.org/10.1145/3639478.3640034