Limitations in Using Multiple MIG Instances in a Container

[tunahanertekin]

Hi,

I am curious about how different it is to use multiple MIG instances instead of multiple no-mig GPUs (such as V100) in terms of paralleling, memory sharing etc. I didn't receive same outputs in processes such as:

• Triton Inference Server
  • I started Triton Inference Server with Docker setting the --gpus argument as "device=0,1,2" on an instance that has 4 V100 GPUs. Triton Inference Server distributed the load between 4 GPU devices.
  • I started Triton Inference Server with Kubernetes (using the device plugin) giving 4 MIG instances (1g.10gb) to the container, on an instance that has 4 A100 80gb GPUs. Triton Inference Server used only the device with ID 0.
• Training with PyTorch
  • I started multi-GPU YOLOV7 training in the repository guide on an instance that has 4 V100 GPUs. I set the --device argument as 0,1,2,3 and the training was successful.
  • I launched the same command giving 4 MIG instances (1g.10gb) to the container, on an instance that has 4 A100 80gb GPUs. I tried to set --device argument as 0,1,2,3 or using UUIDs like MIG-XYZ,MIG-ABC, but couldn't start the distributed training. (also the functions under torch.cuda are seeing the physical GPUs instead of MIG instances.)

Any kind of help is appreciated.

[github-actions[bot]]

This issue has become stale and will be closed automatically within 30 days if no activity is recorded.

[jiusi9]

Hi, I have the same issue, if there was a solution?

[deepakdeore2004]

i am seeing same problem where docker shows only GPU, anyone found the issue?

# docker run --rm -it --gpus '"device=MIG-eb5ec582-a562-505c-8bf6-72a255d3360f,MIG-2e532e9b-c8ac-5a44-ab5f-368b2fc09522"' ubuntu nvidia-smi --query-gpu=uuid --format=csv
uuid
GPU-43ff88d4-81a0-5fd6-e5d9-7c0d43abf38e

[klueska]

That is expected if both of your MIG devices are on the same underlying GPU. What is the output of:

docker run --rm -it \
  --gpus '"device=MIG-eb5ec582-a562-505c-8bf6-72a255d3360f,MIG-2e532e9b-c8ac-5a44-ab5f-368b2fc09522"' \
  ubuntu nvidia-smi -L

[deepakdeore2004]

That is expected if both of your MIG devices are on the same underlying GPU. What is the output of:

docker run --rm -it \
  --gpus '"device=MIG-eb5ec582-a562-505c-8bf6-72a255d3360f,MIG-2e532e9b-c8ac-5a44-ab5f-368b2fc09522"' \
  ubuntu nvidia-smi -L

thanks for quick reply, here is the output

# docker run --rm -it \
  --gpus '"device=MIG-eb5ec582-a562-505c-8bf6-72a255d3360f,MIG-2e532e9b-c8ac-5a44-ab5f-368b2fc09522"' \
  ubuntu nvidia-smi -L
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-43ff88d4-81a0-5fd6-e5d9-7c0d43abf38e)
  MIG 1g.20gb     Device  0: (UUID: MIG-eb5ec582-a562-505c-8bf6-72a255d3360f)
  MIG 1g.20gb     Device  1: (UUID: MIG-2e532e9b-c8ac-5a44-ab5f-368b2fc09522)

[klueska]

So it seems to be working as expected then.

[klueska]

Note, however, that even if oyu have multiple MIG instances available to your container CUDA will only use the first one it discovers. There is no support (yet) for running a single CUDA context with multiple MIG devices.