Suggestion : Remove infrastructure from Ripsaw

Currently Ripsaw attempts to manage other operators, which has been a struggle.

Why did we have Ripsaw launch other operators?

We wanted to have the application be managed via the Operator framework for testing the workload, and to provide an end to end solution.

What is wrong with this approach?

It causes complexity that we continuously have to work around. Broken CI, which slows down our ability to accept PRs

What can we do to replace/remove this dependency for infrastructure?

Instead of Ripsaw managing other operators, we could have our CI/Testing use statefulsets to deploy applications like mongodb, and we simply update our YCSB CR to point at the mongodb deployed with statefulsets.

The approach of having an operator manage/launch another operator might not be necessarily novel, but still waiting to hear from the operator-sdk community on how they tackle an operator launching/managing another operator and its resources.

I vote in favor of it, as there doesn't seem to be a clear/supported way of doing this that's provided by the operator framework

It will be much simpler for us to focus efforts on Ripsaw running only workloads, that's for certain. However, since we have discussed a completely automated benchmark lifecycle, we cannot accomplish that end-to-end without also automating the infrastructure/application under test. If we remain committed to the end-to-end automation, then something will need to take the place of Ripsaw in standing up the application stacks. It seems inefficient to maintain another tool outside of Ripsaw to do this job.

If, on the other hand, we are willing to defer to the Ripsaw user to first deploy and configure the application as a prerequisite, then sure, let's get rid of the infras altogether.

I'm not sure that this alleviates the CI problem much at all, regardless. The pain is in automating the stand-up and tear-down of the infra, and if we still need to automate this inside the CI in order to validate the workload, then we still have most of the same headaches. If we can automate it for the CI, why not leverage it in the benchmark workflow directly?

Whether we deploy infras via other operators or just use pods or stateful sets is a separate question, I think. Testing some stateful applications properly may almost necessitate an operator-based deployment.

It will be much simpler for us to focus efforts on Ripsaw running only workloads, that's for certain. However, since we have discussed a completely automated benchmark lifecycle, we cannot accomplish that end-to-end without also automating the infrastructure/application under test. If we remain committed to the end-to-end automation, then something will need to take the place of Ripsaw in standing up the application stacks. It seems inefficient to maintain another tool outside of Ripsaw to do this job.

If, on the other hand, we are willing to defer to the Ripsaw user to first deploy and configure the application as a prerequisite, then sure, let's get rid of the infras altogether.

My suggestion was to remove the operator aspect. This gets us out of the discussion of which operator we support -- For example today there are two mongodb and postgres Operators on Operatorhub.io. We could still provide a mechanism to deploy postgres/mongodb/etc just without using an operator.

If we wanted to keep some form of infra in Ripsaw to have a end-to-end I think we simply go with the above approach.

I'm not sure that this alleviates the CI problem much at all, regardless. The pain is in automating the stand-up and tear-down of the infra, and if we still need to automate this inside the CI in order to validate the workload, then we still have most of the same headaches. If we can automate it for the CI, why not leverage it in the benchmark workflow directly?

Not suggesting it is the silver-bullet for CI, however it does seem to always be a pain-point (operator managing other operators).

Whether we deploy infras via other operators or just use pods or stateful sets is a separate question, I think. Testing some stateful applications properly may almost necessitate an operator-based deployment.

If an application requires an operator based deployment, I think we defer that application to be deployed outside of Ripsaw. For example, if Couchbase doesn't have a straight-forward way for us to deploy using statefulsets -- similar to the work @aakarshgopi did with mongodb, I suggest we defer the user to Couchbase documentation on operator based deployments, but discuss how we point Ripsaw's YCSB CR at an existing Couchbase DB.

My suggestion was to remove the operator aspect. This gets us out of the discussion of which operator we support -- For example today there are two mongodb and postgres Operators on Operatorhub.io. We could still provide a mechanism to deploy postgres/mongodb/etc just without using an operator.

Alright, this doesn't jive with your issue title here. If we want to keep infras, where reasonable, but not deploy those infras via other operators, then we will still need to maintain some structure to the repo and the CI system accordingly. I'm still thinking about PR #104 and how to get better organized in general, so this decision can affect that.

Ack. Yeah, understand the confusion from title. I'll change that.

If we wanted to keep some form of infra in Ripsaw to have a end-to-end I think we simply go with the above approach.

So we by default rule out any lifecycle automation of a workload that requires an infra that is best deployed via operator? Maybe that won't be an issue in the end, but I'm trying to think ahead to what could get in our way as we expand workloads.

Our intent is to test the performance of the platform. However, if CrunchyDB or name your favorite database operator wants to test their specific implementation of SQL - there is no reason they couldn't use Ripsaw. Our focus shouldn't be the application, it is platform + application, not a specific implementation.

Not suggesting it is the silver-bullet for CI, however it does seem to always be a pain-point (operator managing other operators).

The layered operator is part of the problem, though I think I've mostly solved that with the finalizer and some adjustments to how the catalogsource is deployed. The other and probably bigger problem is that we likely cannot use one basic approach to an operator-based infra for both k8s and OCP 4. This second problem might bite us in a very similar way even if using statefulsets for infra.

If an application requires an operator based deployment, I think we defer that application to be deployed outside of Ripsaw. For example, if Couchbase doesn't have a straight-forward way for us to deploy using statefulsets -- similar to the work @aakarshgopi did with mongodb, I suggest we defer the user to Couchbase documentation on operator based deployments, but discuss how we point Ripsaw's YCSB CR at an existing Couchbase DB.

Fair enough.

I don't want to lose sight of our why.

We are building Ripsaw to run workloads, and it seems we have spent a lot of time on the infrastructure for some workloads.

It will be much simpler for us to focus efforts on Ripsaw running only workloads, that's for certain. However, since we have discussed a completely automated benchmark lifecycle, we cannot accomplish that end-to-end without also automating the infrastructure/application under test. If we remain committed to the end-to-end automation, then something will need to take the place of Ripsaw in standing up the application stacks. It seems inefficient to maintain another tool outside of Ripsaw to do this job. If, on the other hand, we are willing to defer to the Ripsaw user to first deploy and configure the application as a prerequisite, then sure, let's get rid of the infras altogether.

My suggestion was to remove the operator aspect. This gets us out of the discussion of which operator we support -- For example today there are two mongodb and postgres Operators on Operatorhub.io. We could still provide a mechanism to deploy postgres/mongodb/etc just without using an operator.

If we wanted to keep some form of infra in Ripsaw to have a end-to-end I think we simply go with the above approach.

I'm not sure that this alleviates the CI problem much at all, regardless. The pain is in automating the stand-up and tear-down of the infra, and if we still need to automate this inside the CI in order to validate the workload, then we still have most of the same headaches. If we can automate it for the CI, why not leverage it in the benchmark workflow directly?

Not suggesting it is the silver-bullet for CI, however it does seem to always be a pain-point (operator managing other operators).

Whether we deploy infras via other operators or just use pods or stateful sets is a separate question, I think. Testing some stateful applications properly may almost necessitate an operator-based deployment.

If an application requires an operator based deployment, I think we defer that application to be deployed outside of Ripsaw. For example, if Couchbase doesn't have a straight-forward way for us to deploy using statefulsets -- similar to the work @aakarshgopi did with mongodb, I suggest we defer the user to Couchbase documentation on operator based deployments, but discuss how we point Ripsaw's YCSB CR at an existing Couchbase DB.

It will be much simpler for us to focus efforts on Ripsaw running only workloads, that's for certain. However, since we have discussed a completely automated benchmark lifecycle, we cannot accomplish that end-to-end without also automating the infrastructure/application under test. If we remain committed to the end-to-end automation, then something will need to take the place of Ripsaw in standing up the application stacks. It seems inefficient to maintain another tool outside of Ripsaw to do this job. If, on the other hand, we are willing to defer to the Ripsaw user to first deploy and configure the application as a prerequisite, then sure, let's get rid of the infras altogether.

My suggestion was to remove the operator aspect. This gets us out of the discussion of which operator we support -- For example today there are two mongodb and postgres Operators on Operatorhub.io. We could still provide a mechanism to deploy postgres/mongodb/etc just without using an operator.

If we wanted to keep some form of infra in Ripsaw to have a end-to-end I think we simply go with the above approach.

I'm not sure that this alleviates the CI problem much at all, regardless. The pain is in automating the stand-up and tear-down of the infra, and if we still need to automate this inside the CI in order to validate the workload, then we still have most of the same headaches. If we can automate it for the CI, why not leverage it in the benchmark workflow directly?

Not suggesting it is the silver-bullet for CI, however it does seem to always be a pain-point (operator managing other operators).

Whether we deploy infras via other operators or just use pods or stateful sets is a separate question, I think. Testing some stateful applications properly may almost necessitate an operator-based deployment.

If an application requires an operator based deployment, I think we defer that application to be deployed outside of Ripsaw. For example, if Couchbase doesn't have a straight-forward way for us to deploy using statefulsets -- similar to the work @aakarshgopi did with mongodb, I suggest we defer the user to Couchbase documentation on operator based deployments, but discuss how we point Ripsaw's YCSB CR at an existing Couchbase DB.

Our intent is to test the performance of the platform. However, if CrunchyDB or name your favorite database operator wants to test their specific implementation of SQL - there is no reason they couldn't use Ripsaw. Our focus shouldn't be the application, it is platform + application, not a specific implementation.

So to my question about what to do with the work already put into the existing infras... Do we maintain the work somehow, or scrap it altogether. I see three approaches:

Scrap it. We can reference the work in forks and dev branches as needed.
Keep it in Ripsaw, but in an "unsupported" or "experimental" structure of some kind that bypasses CI.
Move it into something like SNAFU for reference outside of ripsaw but inside of cloud-bulldozer.

As agreed on the call, we've removed all infra from Ripsaw. To ensure that the work we put into, can still be made use of and leveraged, I created an infra-operator codenamed cloud-builder. This has all the roles and work related to infra. I've also tried to test mongo and it worked fine. I also tried postgres on minikube but ran into some issues.

closing this as #122 got merged and removed the current-infra that we'd

cloud-bulldozer / benchmark-operator