rough draft of horizontal-ish scaling

[shawkins]

@k-wall @rareddy this is the rough draft of FSO handling the scaling aspects mentioned on the horizontal https://issues.redhat.com/browse/MGDSTRM-11100 and vertical https://issues.redhat.com/browse/MGDSTRM-9976 issues.

This assumes three resource scenarios:

• If no relevant managedkafka, then 0 replicas
• If more than 30 brokers (chosen arbitrarily at this point), then use what's originally in the Deployment with the understanding that this will be set for the developer/trial scenario
• Otherwise use the default upstream resources, which uses a much lower cpu request.

It doesn't appear that having 0 replicas will be problem for OLM, however OLM does double check the hash it creates https://github.com/operator-framework/operator-lifecycle-manager/blob/dac8182eb62acc1cb489d17ccc34f243f43d4f94/pkg/controller/install/deployment.go#L285

If it doesn't match for the operations in https://github.com/operator-framework/operator-lifecycle-manager/blob/783bebf6d4811c0b36eabaa5e58a05a000a1dbfc/pkg/controller/operators/olm/operator.go then the csv won't transition properly through its phases correctly - logging errors or worse.

It does not seem like a good idea to go down the path of copying the hash functionality in Java, nor trying to time our updates to the deployment to match expectations with the OLM operator. So this is likely a dead end.

[MikeEdgar]

It does not seem like a good idea to go down the path of copying the hash functionality in Java, nor trying to time our updates to the deployment to match expectations with the OLM operator. So this is likely a dead end.

Patching the deployment(s) within the CSV might be viable. I don't think OLM will attempt to modify or restore the CSV back to the version unpacked from the bundle.

[shawkins]

Patching the deployment(s) within the CSV might be viable.

Thanks for the suggestion to move things up a level.

I don't think OLM will attempt to modify or restore the CSV back to the version unpacked from the bundle.

At least locally it does not seem to. This will put the CSV back into the installing phase temporarily - are you aware of any monitoring or other reason that would be an issue?

[MikeEdgar]

This will put the CSV back into the installing phase temporarily - are you aware of any monitoring or other reason that would be an issue?

It's possible that during the initial install the OCM status for the addon would take longer than otherwise to become Ready, but it won't impact readiness to provision a Kafka from KFM's perspective since it relies on the MKA's status for a Hybrid installation. We shouldn't have any monitoring of that at the service level.

[shawkins]

@k-wall @MikeEdgar @rareddy here's an update of the draft using the csv modification approach. I've also narrowed it to just the horizontal / replica manipulation. The resource discussion will continue on https://issues.redhat.com/browse/MGDSTRM-9976 and https://issues.redhat.com/browse/MGDSTRM-11132 but if this pr seems acceptable, we can reintroduce this approach as an alternative as well.

[sonarcloud[bot]]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
4 Code Smells

8.5% Coverage
0.0% Duplication