kruize project 2 - Dedicated project cluster (followup #580)

[schwesig]

follow up from

• https://github.com/nerc-project/operations/issues/580
• PU costs would come from Red Hat's current funding allocation because this is the type of work we were expecting to do as part of the "beta" OpenShift AI cluster work.

Details for this issue

• [ ] https://github.com/CCI-MOC/ops-issues/issues/1329
• [ ] adding the overlay to the config repo

This project needs to run in a dedicated test cluster because

• Kruize is not recommended for production
• and because they require more recent versions of software than we have in production. OpenShift 4.13.29 and OpenShift AI 2.8.0
• the heavy workload testing can interfere with other projects
• the workload of other projects can interfere with the test results
• the team needs more global rights to slice the GPU nodes, which can interfere with other projects and configs

---

Project Overview

This research project focuses on optimizing GPU infrastructure usage through Kruize, a platform that tracks GPU usage for each container. By integrating with OpenShift Observability (Prometheus) and using cost and performance models, Kruize provides recommendations for GPU limits. These recommendations can be enforced by GPU time slice schedulers like Run:ai to enhance GPU utilization, aiming to lower costs and improve performance.

Goals:

Install Kruize with OpenShift AI to observe and model resource usage. Provide better resource usage defaults and configuration tuning for improved performance and cost efficiency.

Steps:

0. Create users, project, and resources on NERC (done in https://github.com/nerc-project/operations/issues/580)
1. Create an interim solution until a dedicated project cluster is available (https://github.com/nerc-project/operations/issues/623)
2. Create a dedicated project cluster (THIS https://github.com/nerc-project/operations/issues/624)
3. Support the tests: Running for 90+ days, switching GPU types, etc. (https://github.com/nerc-project/operations/issues/625)
4. Closing the project: remove/archive cluster, GPU allocations, etc. (https://github.com/nerc-project/operations/issues/626)

CC

Dominika Oliver - doliver@redhat.com Rebecca Whitworth - rsimmond@redhat.com Dinakar Guniguntala - dgunigun@redhat.com

@ddoliver @rebeccaSimmonds19 @dinogun @shekhar316 @bharathappali @bhanvimenghani @kusumachalasani

@dystewart @schwesig @Milstein @tssala23

[schwesig]

waiting for? NESE? Template? ?

[tssala23]

@schwesig For the cluster I am bring up I will be adding the overlay to the config repo and applying it to the cluster today. The NESE storage request has been completed however Hakan is currently working on accessing NESE from MOC network https://github.com/CCI-MOC/ops-issues/issues/1329 though I do not think that will take long. CC @hpdempsey

[schwesig]

@schwesig For the cluster I am bring up I will be adding the overlay to the config repo and applying it to the cluster today. The NESE storage request has been completed however Hakan is currently working on accessing NESE from MOC network CCI-MOC/ops-issues#1329 though I do not think that will take long. CC @hpdempsey

@tssala23 thanks for the update. Just wanted to know where we are at.

[schwesig]

Timeline for the testing

• we plan to provide recommendations for short, medium and, long term usage, which currently translates to
  • 1 day
  • 7 days
  • 15 days of usage,
  • this can, however, change to custom definitions, such as long term = 90 days
• This means that we will have to run loads for at least 15 days continually (better 90), so that we can gather metrics for these periods and provide corresponding recommendations So we have a test bench that can run continually for 15 days, generate various benchmark load conditions, and gather metrics during the entire period.
• We then use those metrics to analyze and provide recommendations

Wishlist from the Research Team

Desired GPUs:

• V100
• A100
• H100

Definition of GPU Performance:

• Low-Performance GPUs: Used throughout the quarter
• High-Performance GPUs: Used intensively for a few weeks during the quarter
  • A100 is the staple (and minimum) GPU needed for most testing
  • H100 will be used sparingly whenever available

GPU/Node Requirements:

• A single cluster with a minimum of 3 GPU nodes (minimum configuration)
• Ideally, two such clusters
• One desired configuration: a single node having 4 GPU cards connected using NVLink

[schwesig]

General information/current status:

V100 : easily available and up for long-term usage A100 : available, but allocation must be reduced to extensive testing times and get only involved when needed H100 : not yet available, and the timeline is not stable enough to plan for the tests yet