follow up: New NVIDIA A100 GPUs - Quality Test

nice to have/ follow up for https://github.com/nerc-project/operations/issues/482 feel free to participate in this testing, sharing experiences and results. if help is wanted/needed for observing the tests... contact @schwesig

[x] We are planning to conduct a quantity test for the newly installed NVIDIA A100 GPUs by spinning up 200 RHODS images with GPU claims.

We are planning to conduct a quality test for the newly installed NVIDIA A100 GPUs by running a newly developed "Tensorflow Jupyter CUDA" image, designed to test the computing power within our OpenShift AI environment; focusing on their performance and compatibility. This test will utilize the new Tensorflow Jupyter CUDA image with 02_model_training_basics.ipynb.

This test does not need to be exclusive for this image/script. If anything is missing or there are new scripts or images useful, feel free.

Test Objectives:

Ensure the stability and performance when utilizing the GPUs.
Verify the compatibility and stability using Tensorflow Jupyter CUDA image.

Test Environments :

[x] On Test Cluster
- [x] basic, single workload (done in https://github.com/nerc-project/operations/issues/482)

follow up, nice to have:

[ ] On Test Cluster
- [ ] parallel workloads (2, 3, 4)
[ ] On Prod Cluster, ideally before classes start again (03/18/2024)
- [ ] basic, single workload
- [ ] parallel workloads (2, 3, 4)

Procedure:

Conduct a series of TensorFlow jobs to test GPU performance.
Monitor system stability, noting any crashes, errors.
Deploy the "Tensorflow Jupyter CUDA" image on the Prod Cluster.
Repeat the testing procedure on the Prod Cluster, ensuring consistency and reliability on different clusters.
Document any issues encountered and the outcomes of the tests for both clusters.

This quality test aims to confirm that the new NVIDIA A100 GPUs are working and can be used for upcoming classes and projects.

nerc-project / operations