Closed TheAggressive closed 2 years ago
@monopole Can you take a look at this?
I have run into this too. I think it's due to the way in which CRDs get applied to the cluster. The CRD is applied, then the cluster accepts the kinds and makes them available. But there is a lag in that happening. So your other resource that requires the kind provided by the CRD can't be applied at the time.
If you apply the CRD and the sealed secret in two steps - the process will work - not sure how to make this work in one go right now.
based on @mikee's comment this sounds like an issue with kubectl/kubernetes and not kustomize?
@TheAggressive @johscheuer can you please try just the build step here? This may be a kubectl apply
issue as opposed to a kustomize build
.
I think the problem might be that the CRD is being created AND used in the same call to kubectl apply
.
kubectl apply
won't wait for the CRD to be available before attempting to use it, which results in the error:
error: unable to recognize "STDIN": no matches for kind "SealedSecret" in version "bitnami.com/v1alpha1"
You can tell this is the case because if you just wait a few seconds and run it again, it will succeed.
You need to create the CRD in one step, wait for it to exist, then use the CRD in a second step.
I think the problem might be that the CRD is being created AND used in the same call to
kubectl apply
.
kubectl apply
won't wait for the CRD to be available before attempting to use it, which results in the error:error: unable to recognize "STDIN": no matches for kind "SealedSecret" in version "bitnami.com/v1alpha1"
You can tell this is the case because if you just wait a few seconds and run it again, it will succeed.
You need to create the CRD in one step, wait for it to exist, then use the CRD in a second step.
that is what I've experienced in testing.
that is what I've experienced in testing.
sorry @mikee I didn’t see your previous comment before I commented. Yes, I agree with your assessment of what is happening.
@TheAggressive @mikee What if you remove the remote controller.yaml from the kustomization resources and do something like this instead:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.13.1/controller.yaml
kubectl wait --for condition=established crd sealedsecrets.bitnami.com
kustomize build . | kubectl apply -f -
This should create the crd, wait for it, then apply the kustomize output
@TheAggressive @mikee What if you remove the remote controller.yaml from the kustomization resources and do something like this instead:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.13.1/controller.yaml kubectl wait --for condition=established crd sealedsecrets.bitnami.com kustomize build . | kubectl apply -f -
This should create the crd, wait for it, then apply the kustomize output
That works - what I've done is just split my kustomize into two overlays (using fluxcd). The "second" layer that depends on the CRD may fail the first time, but will eventually reconcile once the CRDs are accepted
This is related to https://github.com/kubernetes-sigs/kustomize/issues/3794 / https://github.com/kubernetes-sigs/kustomize/issues/3913 in that it involves sequencing resources for deploy purposes. As discussed there and above, it is possible for Kustomize users to manually ensure that a given CRD appears before a related CR in the resource list output (using FIFO ordering), but Kustomize is not involved in the apply operation and cannot prevent the race between the cluster-side acceptance of the CRD and the attempted creation of the CR. As @brianpursley has suggested, the deploy must be separated into two phases to prevent this, although @mikee is also correct that simply retrying will eventually work.
/close
@KnVerey: Closing this issue.
For anyone landing here, MY issue (using helm/helmfile) was to add:
`--include-crds
explicitly to the template/upgrade/apply commands I use in our CD pipeline.
For ArgoCD + helmfile, it looks like this now:
server:
logLevel: warn
serviceAccount:
create: true
configEnabled: true
config:
kustomize.buildOptions: --load-restrictor LoadRestrictionsNone
cluster.inClusterEnabled: "true"
ignoreAggregatedRoles: "true"
timeout.reconciliation: 30s
configManagementPlugins: |
- name: helmfile
generate:
command: ["/bin/sh", "-c"]
args: ["helmfile template --include-crds"]
Thank you @armenr! It's working with external-secrets-operator:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
helmCharts:
- name: external-secrets
repo: https://charts.external-secrets.io
releaseName: external-secrets-operator
namespace: vault
version: 0.9.13
includeCRDs: true
resources:
- cluster-secret-store.yaml
I'm not sure how to go about instantiation of a resource and then applying a CRD(?) of that resource in a fresh cluster to setup the cluster.
Example:
But once the kustomize build is ran and applied
kustomize build . | kubectl apply -f -
I see that thesealed-secret.yaml
wasn't applied and no secret was generated from the sealed-secret based on the followingkustomize build . | kubectl apply -f -
output:unable to recognize "STDIN": no matches for kind "SealedSecret" in version "bitnami.com/v1alpha1"
So is there a proper way to get this working? I was looking into
nameReference
in aconfigurations:
section of thekustomization.yaml
but not sure if that will give the desired result?Thank you!