Create roadmap for planning RHOAI updates - Draft 1

[dystewart]

Closing https://github.com/nerc-project/operations/issues/528 and replacing with this issue.

Seeing as we are well behind the most recent releases of openshift ai we will want to plan out which versions we wnat to stop and test new features on. Here is the release history in the github repo for starters: https://github.com/opendatahub-io/opendatahub-operator/releases

We are currently at version 1.33.0 in production and test clusters.

A tentative roadmap:

Prereqs [April 25 - May 3] Resolution of the following issues:

• [x] 537 Make test cluster match production cluster software versions
• [x] 527 Update old OpenShift Test cluster from 4.13 to 4.15

Phase1 [April 25th - May 3rd]: determine the RHOAI and RHEL versions to install

Our RHOAI install plan is shaping up like this v1.33.0 -> v2.1.0 -> v2.4.0
(Our OLM wants to update directly to 2.4 but seeing as starting at version 2.1 we have big change lists it's not a bad idea to make sure all our ope stuff is still working at v2.1.0). We want to make it to at least v2.5 to gain features like nbgitpuller/acceleratorprofile so let's reassess after these couple bumps in version.

• For compatibility with GPU AcceleratorProfile (used for MIG configuration) and Jupyter notebook git puller, select RHOAI version 2.5 or later.

Phase2 [May 3rd - May 10th]: install and config RHOAI cluster

• [x] Update subscription/csv of rhoai operator in test cluster to initiate updates (updates for rhoai currently require manual approval before kicking off)
• [x] Connect to observability cluster

Phase3 [May 10th - TBD]: #562 test the functionality of the new cluster and explore new features

• Classes:

  • [ ] Test rhods-notebooks namespace to ensure it provides instance isolation
  • [ ] Integrate the latest versions of packages into the OPE base image
  • [x] Scalability tests on jupyter containers
  • [x] Test Jupyter Notebooks functionality with ope test-book

• GPU and AI:

  • [ ] Test Python AI tools rebuilt wheels on the new OpenShift cluster. For details, refer to the ongoing project: Rebuilding the Wheel.
  • [ ] Once GPUs are ready, test MIG settings with GPU AcceleratorProfile
  • [ ] Test RHEL AI

[hpdempsey]

I asked Gagan to help coordinate this with the OpenShift AI group, because he was already working with them on running pre-GA release versions of RHOA anyway. I can't assign him to the ticket (yet) because he isn't a member of this repo.

[hpdempsey]

I need a rough cut at this NLT 4/29 because we have a program call with the RHOAI team on 4/30.

[DanNiESh]

@hpdempsey I've updated a rough roadmap, please remind me if I'm missing anything. @dystewart Could you add more details to Phase 2 install cluster?

[hpdempsey]

You mean you updated the roadmap in this issue description @DanNiESh ?

[hpdempsey]

I found out that we will also need a new version of RHEL for this (discuss details offline), so that's another step of integration and test.

[DanNiESh]

You mean you updated the roadmap in this issue description @DanNiESh ?

Yes. I updated it in the issue description.

[dystewart]

@hpdempsey @DanNiESh I've made the beginning of an update plan in Phase 1 above and left a small addition to Phase 2 re actual upgrade install info

[joachimweyl]

@gagansk please accept the GitHub invite so I can add you to this issue.

[joachimweyl]

@dystewart what are the next steps for this issue?

[DanNiESh]

Close it as draft 1. If we have new plans that need to add to the roadmap, we will create a v2 draft and copy & paste everything here.