testkube operator integration as a flux controller

[gberche-orange]

Is your feature request related to a problem? Please describe.

testkube-operator does not seem to integrate yet well with flux ecosystem, for flux consumer projects :

• the source (git repo) are fetched independently (including webhooks) :
  • from testkube testsource section
  • from flux source controller https://fluxcd.io/flux/components/source/
• testkube does not yet support all notifications that flux could offer through the notification controller https://fluxcd.io/flux/components/notification/
• testkube operator metrics are similar yet inconsistent with flux controller metrics https://fluxcd.io/flux/guides/monitoring/#metrics

Describe the solution you'd like

testkube operator to also be integrated as a flux extension, see https://fluxcd.io/ecosystem/#flux-extensions

• testkube operator would leverage flux source controller for fetching sources, see https://fluxcd.io/flux/gitops-toolkit/source-watcher/

  In this guide you’ll be developing a Kubernetes controller with Kubebuilder that subscribes to GitRepository events and reacts to revision changes by downloading the artifact produced by flux source-controller.

• testkube operator would integrate so that the flux notification controller https://fluxcd.io/flux/components/notification/ would be able to emit testkube notifications
• testkube artifact storage features would be generalized into a new flux sink-controller that would in the future allow storing artifacts beyond object storage to other sinks: git repo, OCI image...

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context, CLI input/output, etc. about the feature request here.

[vsukhin]

Thank you @gberche-orange for such a thoughtful feature request. I wish, we would have all of them like yours :) Testkube definitely needs to be as much as possible GitOps complaint, we did it for ArgoCD, add triggers, but need more I will ask @TheBrunoLopes for prioritisation of this request

[TheBrunoLopes]

Hello @gberche-orange awesome feature request. thank you for all the fine details. I can definitely tell that having a great integration with Flux is something that falls pretty much in the vision for Testkube and our principle of following the GitOps mindset. Give us a few days to review it, we'll most likely tackle it in the next release.

[cedricmckinnie]

Here's an idea for a simple way to integrate the two in the short-term. If you allow TestTriggers to trigger tests on CustomResources, it would be possible to execute a Test on HelmRelease/Kustomization Conditions i.e.

apiVersion: tests.testkube.io/v1
kind: TestTrigger
metadata:
  name: testtrigger-example
spec:
  resource: <HelmRelease|Kustomization>
  resourceSelector:
    namespace: foo-ns
    name: my-helmrelease
  event: <created|modified|deleted>
  conditionSpec:
    timeout: 100
    conditions:
    - reason: ReconciliationSucceeded
      status: "True"
      type: Ready
    - reason: UpgradeSucceeded
      status: "True"
      type: Released
  action: run
  execution: test
  testSelector:
    name: test-to-run
    namespace: testkube

It's wonderful that TestTriggers already support labelSelectors as well.

This would provide a good intermediate step to allow people to start using it and perhaps offer more suggestions for improvement. Seems like it could be doable in a reasonable amount of time as well.

I'm very interested in this functionality and would be happy to share more ideas on this if needed!

[gberche-orange]

Give us a few days to review it, we'll most likely tackle it in the next release.

@TheBrunoLopes @vsukhin could you please share your current thoughts about fluxcd gitops toolkit usage by testkube ?

Here is a refinement of possible UX for integrating flux devops toolkit into testkube:

Test sources https://docs.testkube.io/articles/test-sources/ get extended to reference a fluxcd source CR ref

apiVersion: tests.testkube.io/v1
kind: TestSource
metadata:
  name: gitops-fluxcd-repo-testkube-source
spec:
  sourceRef: # exclusive with repository
    # GitRepository  or any of flux source controller at https://fluxcd.io/flux/components/source/ such as:                                                                                                                                                                                              
    # Bucket
    # OCIRepository 
    kind: GitRepository 
    name: my-gitops-repo 

During test execution, the runner init container would fetch the cached source tar.gz archive from the source controller rest endpoint (see https://fluxcd.io/flux/gitops-toolkit/source-watcher/) and then copy into the /data/source mount point (different naming than /data/repo to reflect that this might include bucket content).

The fluxcd source controller notifications would also support:

• the source change trigger feature suggested into https://github.com/kubeshop/testkube/issues/4007
• flux events triggers, similar to suggested into #3712

This would also likely increase tk scalability by reducing load on remote upstream sources, such as gitrepo serving monorepos (albeit existing tk shallow clones and sparse checkout), and leverage the existing extensive fluxcd sourcecontroller caching investments.

In terms of market mindshare, this integration is likely to bring to testkube a large body of opensource flux users including enterprise customers. They're likely to express additional enterprise requirements that might help fuel the great testkube development efforts.