Open rajewluk opened 1 year ago
hey @rajewluk we started with this approach. it was a nice thing, but the prroblem was Helm didn't update CRD witj latest changes in this case. Might be, Helm was improved since that time, for @ypoplavs and @dejanzele t check
Understood, but the current approach makes the use of testkube in gitops approach very difficult and unreliable on upgrade.
Possibly, making them optional in the templates folder (flag in the values) and having them in the crds folder could help to solve it. First, they would get installed from crds folder, then would get updated from templates (if enabled, the helm app case). In ArgoCD the templates could be basically disabled as those from crds would get synced always by ArgoCD.
yes, let's review this issue again and see if we can do it better @ypoplavs
Any progress on this? This seems to still be the case when using Flux GitOps:
Helm install failed for release testkube/testkube with chart testkube@2.0.6: failed post-install: unable to build kubernetes object for deleting hook testkube/charts/testkube-api/templates/testworkflows/builtin-templates/distribute.yaml: resource mapping not found for name: "distribute--evenly" namespace: "" from "": no matches for kind "TestWorkflowTemplate" in version "testworkflows.testkube.io/v1"
ensure CRDs are installed first
hey, @devantler it should be realted to test workkflow template post hook installation, which should happen after CRD one. @tkonieczny and @emil2k can clarify
Awesome, looking forward to hear from them. And just for context, I am currently not setting any Helm values:
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: testkube
spec:
interval: 1m
chart:
spec:
chart: testkube
version: 2.0.6
sourceRef:
kind: HelmRepository
name: testkube
# https://github.com/kubeshop/helm-charts/blob/develop/charts/testkube/values.yaml
values: {}
I made it work for now by deploying the CRDs prior to deploying the Helm chart.
# Temporary solution!!!
# This Flux Kustomization deploys the CRDs for the Testkube Helm chart.
# The CRDs are copied from https://github.com/kubeshop/testkube-operator/tree/main/config/crd/bases
# For them to work you need to add the following labels and annotations to the CRDs:
#
# ```yaml
# annotations:
# meta.helm.sh/release-name: testkube
# meta.helm.sh/release-namespace: testkube
# labels:
# app.kubernetes.io/managed-by: Helm
# ```
#
# And you also need to ensure the testkube.yaml Flux Kustomization depends on this one.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: testkube-crds
namespace: flux-system
labels:
app.kubernetes.io/sops: enabled
app.kubernetes.io/post-build-variables: enabled
spec:
interval: 1m
targetNamespace: testkube
sourceRef:
kind: OCIRepository
name: flux-system
path: testkube/crds
prune: true
wait: true
thank you @devantler we didn't have time to investigate it so far
Testkube operator CRDs are placed in the templates folder of the operator helm chart. In consequence, when testkube fails and webhook service is not available the CRD and its CRs cannot be deleted. Basically, it makes Flux/ArgoCD tools work badly with testkube since they try to delete CRD when CRs are not deleted and that gets stuck if testkube services are already not available (as they get deleted in the random order along with other types of resources). To avoid it, CRDs should be placed in the crds folder of helm chart, as helm documentation states.