Spike: Revisit decision about creation of OPE cluster

[msdisme]

The decision not to create a separate OPE cluster was made a few weeks ago, the thinking being that it was better to add additional nodes to the current cluster if needed. Since then, a large class has been added (3, up from original 2 classes). This issue will revisit whether we should have a second OPE cluster as a fallback.

• [x] Discuss in this issue the pros and cons
• [x] Decide whether to set up an additional cluster based on that discussion
• [x] The nodes are still available , so either open an issue to stand up the cluster if that is the decision or create a new issue with a repurposed use for those nodes.

[dystewart]

Update from yesterday's ope meeting:

We will move forward with standing up a cluster as a backup for ope courses.

In short:

Pros:

• We will no longer need an AWS cluster as a backup
• We won't be paying a third party (AWS) for usage and relying on their infrastructure

Cons:

• We need to manually deploy and configure this cluster

I'm going close this, and open another issue to track setting up this cluster.

[larsks]

@dystewart @joachimweyl I'd like to propose closing this issue; I think we're happy with the performance of the production cluster so far, and while we plan to deploy another cluster to act as a RHOAI test cluster, that use is somewhat orthagonal to this issue.

[hpdempsey]

This issue needs to remain open. We have requests from BU for this, and I believe the OpenShift AI tests can simultaneously work as an option to satisfy those requests, even though the releases won't be equivalent.

[hpdempsey]

Once we have finished the basic testing with the classes on equivalent software to production, then we can save that configuration in case we need to spin up a production backup during the semster, and then close this one and proceed to further testing.