Getting started with EFA on EKS

Continuing the repo in aws-samples/eks-efa-examples this repo provides an example using managed nodegroups to instantiate a EKS cluster with GPUDirectRDMA on p4d instances.

EFA Basics

Please read https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html for basic knowledge of EFA

Kubernetes and EKS Basics

This document is assuming user has basic knowledge on Kubernetes and Amazon EKS

Abstract

This document will introduce the user to create an EKS cluster with p4d.24xlarge backed nodegroup with EFA and GPUDirect RDMA, to run an example NCCL-Test for Multi-node NCCL Performance via EFA. This workflow will also provide a template for distributed deep learning training on EKS using EFA.

Step 1: Create EKS cluster

Create an empty EKS cluster via eksctl. In the later steps, the nodegroup would be created then join the EKS cluster.

eksctl create cluster --region us-east-1 --without-nodegroup --vpc-public-subnets ${subnets} 

Step 2: Create Launch Template, EFA Security Groups and Placement Groups

Specific to executing ML workloads and other applications that take advantage of GPUDirect RDMA backed with 4x100 Gbps EFA. Key requirements for EFA networking are creating the EFA specific security groups and placements groups as well launching the nodegroup in a single Available Zone. To simplify this lift we have packaged all the requirements in a cloudformation script which will create a launch template. This launch template can be used in a managed nodegroup.

In AWS CloudFormation Console upload the cloudformation script in cloudformation/efa-p4d-managed-nodegroup.yaml apply fill in the relevant details:

• In ClusterControlPlaneSecurityGroup, use the control plan security group (the name contains control plan) which is automatically created by EKS when user created via eksctl.
• In NodeImageIdSSMParam, ensure it is using Amazon EKS-optimized accelerated AMI: /aws/service/eks/optimized-ami/1.19/amazon-linux-2-gpu/recommended/image_id.
• In Subnetid, it is recommended to put in a private subnet. To select a subnet where p4d instances are available in a region you can run this comman:

  aws ec2 describe-instance-type-offerings --location-type availability-zone --filters Name=instance-type,Values=p4d.24xlarge --region <region>

When the cloud formation is in CREATE COMPLETE continue to Step 3.

Step 3:

To complete the EKS setup we can now associate the created launch template with the EKS cluster. All aws cli command line flags can be retireved from the cloudformation Outputs sections

aws eks create-nodegroup --cluster-name <cluster_name> --nodegroup-name <node_group_name> --node-role <role_arn> --subnets <subnet_id> --launch-template id=<launch_template_id>,version=1

After the nodegroup is created, check the eks cluster for the nodes to join the cluster.

watch kubectl get nodes

The nodes should join after a few moments

NAME                                          STATUS   ROLES    AGE   VERSION
ip-172-31-65-211.us-west-2.compute.internal   Ready    <none>   65m   v1.19.6-eks-49a6c0
ip-172-31-89-239.us-west-2.compute.internal   Ready    <none>   65m   v1.19.6-eks-49a6c0

Step 4: Install AWS EFA EKS Plugin, NVIDIA Device Plugin, Kubeflow MPIOperator

In order to present the underlying pod(s) with the EFA RDMA devices we are providing a device plugin you can install TODO: Provide a permament home EFA EKS plugin

kubectl apply -f https://raw.githubusercontent.com/aws-samples/aws-efa-eks/main/manifest/efa-k8s-device-plugin.yml

Next apply the NVIDIA device plugin DaemonSet to present the pods with the underyling GPU devices.

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.8.2/nvidia-device-plugin.yml

For the NCCL tests we will now apply the Kubeflow MPIOperator which will help manage the pod networking.

git clone https://github.com/kubeflow/mpi-operator
cd mpi-operator
kubectl apply -f deploy/v2beta1/mpi-operator.yaml

Step 5: Run the Multi-node NCCL Performance Test on the EKS cluster for verifying GPUDirectRDMA/EFA

To check NCCL Performance with GPUDirectRDMA over EFA, run the standard NCCL performance test that is available on the official NCCL-Tests Repo. The Dockerfile comes with this test already built for both CUDA 11.2 and the latest version of EFA/AWS-OFI-NCCL. You can similarly run Kubernetes job without EFA. Alternatively we provide a docker image available in ECR Public here: public.ecr.aws/w6p6i9i7/aws-efa-nccl-rdma:base-cudnn8-cuda11-ubuntu18.04

HugePages The most important modification required in a Kubernetes job for adopting EFA is configuring and managing Huge Pages as a schedulable resource in the cluster. Currently, nodes with EFA support pre-allocates 5128 2M Huge Pages.

The example to run 2 node NCCL Performance Test is examples/nccl-efa-tests.yaml. In the example NCCL-test job, each worker requested 8 gpus which would allow cluster to allocate two nodes in two P4d instances, and also 256Mi hugepages-2Mi and 8000Mi.

To pass EFA devices into your container so that your code can use them make a resource limit/request in your podspec

resources:
              limits:
                nvidia.com/gpu: 8
                hugepages-2Mi: 5120Mi
                vpc.amazonaws.com/efa: 4
                memory: 8000Mi
              requests:
                nvidia.com/gpu: 8
                hugepages-2Mi: 5120Mi
                vpc.amazonaws.com/efa: 4
                memory: 8000Mi

Create NCCL-tests job via command:

$ kubectl create -f examples/nccl-efa-tests.yaml
mpijob.kubeflow.org/nccl-test-debug created

The MPIOperator creates a launcher pod and 2 workers pods (one on each node)

NAME                            READY   STATUS              RESTARTS   AGE
nccl-tests-efa-launcher-wzr8j   0/1     Init:0/1            0          5s
nccl-tests-efa-worker-0         0/1     ContainerCreating   0          5s
nccl-tests-efa-worker-1         0/1     ContainerCreating   0          5s

The whole log returned by command:

$ kubectl logs -f nccl-tests-efa-launcher-wzr8j

Security

See CONTRIBUTING for more information.

License

This project is licensed under the MIT-0 License.