GPU Integration

[fbaig]

Problem Larger (especially ML/AL) workloads require access to GPUs for parallel processing. During summer school 2023, GPU instances were made available in the platform backed by Anvil and JetStream GPU nodes. However, keeping these instances running burn a lot of SUs. At the same time, it's difficult to assign a single GPU instances per user.

Potential Solution

• For development and debugging, provide access to small GPU instances to specific users. Will these be dedicated GPU instances?
• For larger testing and scalability, add support for GPU workloads in CyberGIS-Compute. These will mostly be shared GPU instances hosted on the backend HPC environment

Pull Requests ToDo ...

[alexandermichels]

We have a very simple test case working on Keeling with CyberGIS-Compute (https://github.com/alexandermichels/geoai-model-testing.git), but this isn't a full solution. Matching software versions exactly between software and HPC is going to be difficult. It's possible that we could install software stacks with NVIDIA through CVMFS, but CVMFS with Compute is also in a very experimental stage currently. Open to any suggestions.

Edit: the repo I linked has a Dockerfile from an undergrad, but it is not the one we are currently using. This is our current image: https://github.com/cybergis/docker-images/tree/main/cybergis-compute/pytorch

[rkalyanapurdue]

It might be useful to look into Nvidia NGC containers. We have them deployed on Anvil via the module system. I think there might be corresponding docker images we can adapt for Jupyter integration.

However these default images did not seem to work for the summer school team 6 use case, so that might need another look..

[fbaig]

Thread-1: Direct access to GPU (Lead by @nosolls ) Launch GPU instance on the fly only when a user requests it.

• [ ] How long does it take to launch a GPU instance? (Anything less than 5 minutes is great as an initial step)
• [ ] Create an ANSIBLE script to automate the process which will be called when a user requests a GPU instance
• [ ] Automate shutting down the instance if session is idle for a specified time
• [ ] If above works for GPU instances on JetStream, generalize and test for non-JetStream GPU instances

[fbaig]

Thread-2: GPU Access with CyberGIS-Compute (Lead by @alexandermichels )

• [ ] Test CVMFS with GPU software stack

[yirugi]

2/26 Hacking session:

Modified terraform configs for JetStream K8s to add NVIDA GPU support

• The terraform configuration for JetStream K8s has been enhanced to incorporate GPU support, accomplished by integrating the "nvidia-container-toolkit" installation, configuring "containerd" to include the NVIDIA GPU hook, and enabling GPU functionality within Kubernetes.

• Terraform Configuration Repository: https://github.com/nosolls/terraform-js2-k8cluster

• GPU Configuration YAML: https://github.com/nosolls/terraform-js2-k8cluster/blob/main/ansible/gpu.yaml

Created Dockerfile for Jupyter Notebook + NVIDIA + CVMFS

• A unified Dockerfile tailored for Jupyter Notebook usage has been developed. This Dockerfile utilizes the nvidia/cuda base image, augmented with the foundational/base elements of Jupyter, and incorporates I-GUIDE cvmfs images.

  • “nvidia/cuda:11.2.2-cudnn8-devel-ubuntu20.04” base image

  • The CUDA version in Anvil is 11.2.2, and 11.8 in JetStream

  • We could change it to “runtime” image to downsize the image size

• Docker Image Repository: https://hub.docker.com/r/cybergisx/i-guide-gpu

• Dockerfile:https://github.com/cybergis/docker-images/blob/main/jupyter/i-guide-gpu/Dockerfile

[rkalyanapurdue]

Relies on #17 ; will need to upgrade the Kubernetes version before we can use TerraForm and Ansible to attach a new GPU node to the cluster. Older versions of Kubernetes are no longer accessible from the apt repos.