search

pulumi / pulumi-kubernetes-operator

A Kubernetes Operator that automates the deployment of Pulumi Stacks
Apache License 2.0
221 stars 55 forks source link

Error: "expected a non-empty string in .status.artifact.url" #345

Closed lukehoban closed 1 year ago

lukehoban commented 1 year ago

Following #344, I recreated the GitRepository and Stack. This time I got many instances of this in logs:

{"level":"error","ts":"2022-10-19T04:41:14.857Z","logger":"controller_stack","msg":"Failed to update Stack","Request.Namespace":"default","Request.Name":"basic-c7099c01","Stack.Name":"lukehoban/basic/docker-desktop","error":"expected a non-empty string in .status.artifact.url","stacktrace":"github.com/pulumi/pulumi-kubernetes-operator/pkg/controller/stack.(*ReconcileStack).Reconcile\n\t/home/runner/work/pulumi-kubernetes-operator/pulumi-kubernetes-operator/pkg/controller/stack/stack_controller.go:345\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214"}

I can see the artifact url is present (at least after I checked):

status:
  artifact:
    checksum: f8fe96008820b8f01e04d3a6fcf9611366dc4413c9c23953fc536d9b8eef9e0a
    lastUpdateTime: "2022-10-19T04:41:14Z"
    path: gitrepository/default/pko-dev-625ef86f/421a9b36cf29f2a3e433018d1959edb63e7af8eb.tar.gz
    revision: main/421a9b36cf29f2a3e433018d1959edb63e7af8eb
    size: 778
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pko-dev-625ef86f/421a9b36cf29f2a3e433018d1959edb63e7af8eb.tar.gz
  conditions:
  - lastTransitionTime: "2022-10-19T04:41:14Z"
    message: stored artifact for revision 'main/421a9b36cf29f2a3e433018d1959edb63e7af8eb'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2022-10-19T04:41:14Z"
    message: stored artifact for revision 'main/421a9b36cf29f2a3e433018d1959edb63e7af8eb'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  contentConfigChecksum: sha256:fcbcf165908dd18a9e49f7ff27810176db8e9f63b4352213741664245224f8aa
  observedGeneration: 1
  url: http://source-controller.flux-system.svc.cluster.local./gitrepository/default/pko-dev-625ef86f/latest.tar.gz
squaremo commented 1 year ago

OK I see. It'll probably bounce off a new source a few times (quickly, but with a backoff), while the source-controller is fetching the artifact. The stack controller checks to see whether the resource is explicitly marked as not ready; but, it can't assume anything while the source has no indication of being ready or not, which will be the case until the source controller has processed it at least once.

It can do a slightly better job at assessing readiness by looking for a "reconciling" condition in the source; but that won't help when the controller hasn't even seen the resource yet.

This would be improved by fixing the Flux source part of #343, because the stack controller would rely on the watch to tell it when the source has changed, rather than retry-with-backoff. In the meantime, we can reduce logs coming from unready sources by requeuing at a fixed period (with the disadvantages that not having a backoff entails).

squaremo commented 1 year ago

Resolved with #348 (watch Flux sources, rather than polling them) and #359 (explicitly check sources for an artifact and wait for a change if there isn't one).

pulumi-bot commented 1 year ago

Cannot close issue without required labels: kind/