[PROPOSAL] Delete unmanaged indextemplates and ISM policies

[cthtrifork]

What/Why

What are you proposing?

All unmanaged index templates and ISM policies should be deleted when reconciling the resources by the operator.

What problems are you trying to solve?

This is to enforce a strict GitOPs approach for these custom resources and is also a convenient way to make it possible for the operator to delete old templates and policies (by omission).

Are there any security considerations?

We will need to create a blacklist for an internal/system templates or policies that should not be affected by this. It should also be configurable and default this behaviour should not be enabled.

What will it take to execute?

The operator should be expanded with an option to verify all custom resources against the list in the opensearch and delete unmatched items.

We would be willing to make a PR for this feature, if is no major concerns. We would add it to the reconciler logic for ISM policies and templates.

[swoehrl-mw]

Hi @cthtrifork. In general I have no problem with this idea. But the implementation could be tricky as the reconcilers only deal with the specific object they are reconciling. Right now there is no place in the operator that has an overview and could do such a cleanup. If you want to give this a go, feel free.

a convenient way to make it possible for the operator to delete old templates and policies (by omission).

Basically the operator already supports this, as it reacts to deletion of CRs in Kubernetes by deleting the object in opensearch. And via the use of finalizers CRs cannot be deleted without the operator knowing about it. So if you use something like FluxCD and delete an indextemplate in git, which triggers Flux to delete it from Kubernetes, the operator should already handle that usecase. But yes, unmanaged objects are currently not covered by the operator.

[cthtrifork]

Ah yes, thanks for reminding me of the behavior of deleting the custom resources!

My naive thought on solving this, would be to get all templates on the opensearch cluster in a list, reconcile the custom resources and remove matches from the list. Finally, delete items still present in the list. The same should be done for the policies.

I have not looked at the code base for a while, but i will do that soon and update the thoughts on the solution design. Any feedback is welcome at any time!

[rkthtrifork]

It seems there is

resourceList := &opsterv1.OpenSearchISMPolicyList{}
err := r.List(ctx, resourceList , opts...)

which I think gets all the resources and we can filter them. I don't know if it would be reasonable to do it in the reconciler for OpensearchCluster or if it should be in a separate reconciler. Just from a quick look at what seems possible

https://sdk.operatorframework.io/docs/building-operators/golang/references/client/#list

[swoehrl-mw]

@rkthtrifork Yes, the list operation is probably the best way to get an overview over all policies currently defined in K8s.

I don't know if it would be reasonable to do it in the reconciler for OpensearchCluster or if it should be in a separate reconciler.

IMO it should be its own reconciler that is called in the opensearchClusterController (just like for example the snapshot repository reconciler). That way it is a clear separation of concerns and better modularization of the code.