Figure out how to make Timoni's modules work with Flux

[stefanprodan]

Currently, Timoni can be used with Flux by building module instances in CI and pushing the resulting Kubernetes manifests to a container registry, from where Flux picks it up and applies the manifests (docs here).

Flux source-controller is compatible with Timoni's modules published to container registries. The source-controller can pull the modules inside the cluster and verify their cosign signatures, what's missing is a dedicated controller that can managed Timoni's instances.

One solution would be to collaborate with @phoban01 and extend https://github.com/phoban01/cue-flux-controller with support for Timoni's instances, where the values would be supplied from Kubernetes Secrets and ConfigMaps.

Another solution would be to create a timoni-controller from scratch.

[squaremo]

One solution would be to [...] and extend https://github.com/phoban01/cue-flux-controller with support for Timoni's instances, Another solution would be to create a timoni-controller from scratch.

The deciding factor might be this: are there any aspirations towards lifecycle management in Timoni? Things like Helm's pre-install and post-install hooks, I mean. If there are, this would end up being more like the Helm controller; but if not, I think it would be keeping in the spirit of the existing Cue controller to teach it about Timoni modules.

[stefanprodan]

The deciding factor might be this: are there any aspirations towards lifecycle management in Timoni? Things like Helm's pre-install and post-install hooks

Yes that's a good point. Currently Timoni doesn't have the same hooks feature like Helm but it does have a feature to group Kubernetes manifests and it can perform a multi-step apply .e.g.

timoni: {

    // Pass Kubernetes resources outputted by the instance
    // to Timoni's multi-step apply.
    apply: prerequisites: [...]
    apply: frontend: [...]
    apply: backend: [...]
    apply: tests: [...]
}

Example here: https://github.com/stefanprodan/timoni/blob/main/examples/redis/timoni.cue#L36-L39

When multiple stages are specified within timoni.apply, Timoni applies them in order, and moves to the next stage only if the previous one becomes ready. A Flux controller for Timoni should have the same behaviour.

[errordeveloper]

Currently, Timoni can be used with Flux by building module instances in CI and pushing the resulting Kubernetes manifests to a container registry, from where Flux picks it up and applies the manifests (docs here).

It's not immediately clear what are the limitations of this approach, it certainly seemed like the most obvious and viable thing to me, to the point that I kind of doubted the nature of Timoni having to have access to the API. Having talked with @stefanprodan on Slack, I did come to a realisation that runtime rendering is highly beneficial, as it will enable faster iteration cycles. E.g. if some values need to change, there is no need for an upstream change in the repo where the config came from and the CI job doing a rebuild, instead values can be tweaked locally. Of course, this use-case can only be catered for by a fast, reliable, and deterministic renderer, but we can assume CUE ticks those boxes. Just a few more thoughts on build time rendering, the first place where it falls apart is a need to have a namespace as a parameter. Some folks might prefer build-time rendering as it can be perceived as a trust boundary, as some parameters can result in security or supportability issues, however, that could probably be addressed by having a policy of some sort that restricts what values can be set at runtime.

[onedr0p]

Maybe there could be a timoni-controller and a TimoniRelease CR that looks similar to the HelmRelease or OCIRepository spec but built for specifically for Timoni?

phoban01/cue-flux-controller doesn't appear to be active (2 commits in the past year) so the project status is unclear and might cause problems on integrating timoni features?

I am concerned with how people will provide timoni values.cue in this case. Say you bundle up a application to share with others, how would they apply their values.cue?

---
apiVersion: v1alpha1
kind: TimoniRelease
metadata:
  name: thing
spec:
  interval: 1m
  url: oci://ghcr.io/some/timoni-app
  ref:
    tag: v1.0.0
  valuesFromFile:
    - path: thing.cue

[hiddeco]

The thing with producing a final artifact to be consumed by the kustomize-controller is that you loose the Timoni feature of "layered" applies.

On the other hand. If you create a controller that also does the apply, you will end up with about 2/3 of the kustomize-controller, except for the Kustomize bits being swapped out for Timoni calls.

With regard to valuesFromFile, I do not see why this could not be a valuesFrom with a Secret reference which contains a values.cue.

[stefanprodan]

@onedr0p Timoni uses a different terminology, instead of Release it's Instance, more on this here: https://timoni.sh/#timoni-instances.

Instead of having a CR for Instance, I would go with Bundles as the only CRD.

[syncharny]

Is it only the phased apply that differentiates the flux pipeline from timoni apply? Could we have timoni build accept query paths so that we could store pre-deplomentsl specs in one folder, and post-deployment specs in another folder, then use flux's job pipeline?

https://fluxcd.io/flux/use-cases/running-jobs/

[emil-jacero]

@stefanprodan Is your approach to developing the runtime features and getting a feeling for the workflow before investing time into developing a flux controller or are there no plans for a controller at the moment?

[artem-nefedov]

If Timoni is to replace Helm, I would at least expect the following (this goes for both Timoni itself and whatever controller will be managing it in cluster):

1. Ability to provide "values" based on schema defined in the module, similar to values with json schema in Helm chart, or variables in Terraform module.
2. Support for pre/post-action hooks. This part doesn't seem to be fully solvable by Timoni's application order, as I don't see how it can trigger pre-delete/post-delete Job for example.

[zs-dima]

@artem-nefedov I mentioned it in the other issue with TimoniRelease example: https://github.com/stefanprodan/timoni/issues/296 However would be nice to see universal solution that will work with ArgoCD as well.

[salotz]

It looks like from the docs that timoni controller is planned, but just wanted to put in my use case.

I like the idea of being able to deploy the flux-aio into customer environments and have it track drift and update helm charts we deploy them (via the Timoni module). The same for Timoni modules would be great.

Because our installations are sort of hands off after we install them having drift detection (those pesky customers messing with things) and automatic updates straight from an OCI registry without customer intervention would probably handle a lot of support requests.

[salotz]

We've been using timoni with great success for most of this year. I'm just curious as to how much more dogfooding would be necessary to make the controller possible to focus on. In our experience there really hasn't been any issues with the interfaces at all. Only minor complaints is on some inconsistency in the CLI command hierarchy which I don't think effects the controller much.

[errordeveloper]

@salotz I think sponsoring the project might be an option to get this moving forward.

[salotz]

@errordeveloper that could be a possibility. However, it seemed there were other blockers I wanted to check in on.