the manifests of the target version 1.9.2 are not available to this release

Describe the bug When using any version of 1.10.x or 1.11.x we keep getting the error that the manifests of the target version (1.9.2) are not available and are trying to upgrade our knative stack.

We have an existing CRD in place that is working in multiple clusters. Knative CRDs are installed, the knative-serving definition is installed but knative-serving itself is not installed and is not installing.

I have used "dive" against the docker image and it can be seen to have 1.9.2's manifest in there. I have set the log level to debug and development mode to true to get more diagnostics but they are limited and no information is available on where it is looking.

Expected behavior It should install 1.9.2.

To Reproduce I am looking for assistance in how to diagnose this issue, I have had no luck in the past diagnosing with kind. I am expecting that I will have to build a custom image to see if I can figure out what is causing the problem.

Knative release version Any version of the Operator 1.10x -> 1.11.x against 1.9.2

Additional context Just looking for some guidance on how to diagnose.

logs from docker image: gcr.io/knative-releases/knative.dev/operator/cmd/operator@sha256:7d489bfcf23d8515a2913e21b6b6e9c1755175b57b0825602a698be2a8b9f6de

{"severity":"INFO","timestamp":"2023-08-17T00:47:12.546937441Z","logger":"knative-operator","caller":"leaderelection/context.go:158","message":"\"knative-operator-746cb8575-l7gpv_8daca7ed-8249-441f-9eee-9a65a18e621b\" has started leading \"knative-operator.knative.dev.operator.pkg.reconciler.knativeserving.reconciler.00-of-01\"","commit":"7ef32df-dirty","knative.dev/pod":"knative-operator-746cb8575-l7gpv"}
{"severity":"INFO","timestamp":"2023-08-17T00:47:12.547226723Z","logger":"knative-operator","caller":"knativeserving/knativeserving.go:107","message":"Reconciling KnativeServing","commit":"7ef32df-dirty","knative.dev/pod":"knative-operator-746cb8575-l7gpv","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"91dc27b9-bd12-49e2-ac09-0586633d5728","knative.dev/key":"knative-serving/knative-serving","status":{"observedGeneration":1,"conditions":[{"type":"DependenciesInstalled","status":"Unknown","lastTransitionTime":"2023-08-17T00:19:44Z"},{"type":"DeploymentsAvailable","status":"Unknown","lastTransitionTime":"2023-08-17T00:19:44Z"},{"type":"InstallSucceeded","status":"False","lastTransitionTime":"2023-08-17T00:20:14Z","reason":"Error","message":"Install failed with message: the manifests of the target version 1.9.2 are not available to this release"},{"type":"Ready","status":"False","lastTransitionTime":"2023-08-17T00:20:14Z","reason":"Error","message":"Install failed with message: the manifests of the target version 1.9.2 are not available to this release"},{"type":"VersionMigrationEligible","status":"True","lastTransitionTime":"2023-08-17T00:19:44Z"}]}}
I0817 00:47:19.356334       1 leaderelection.go:258] successfully acquired lease knative-operator/knative-operator.knative.dev.operator.pkg.reconciler.knativeeventing.reconciler.00-of-01
{"severity":"INFO","timestamp":"2023-08-17T00:47:19.356446972Z","logger":"knative-operator","caller":"leaderelection/context.go:158","message":"\"knative-operator-746cb8575-l7gpv_e0a0ae47-388f-4074-8fcd-5179302c3f8d\" has started leading \"knative-operator.knative.dev.operator.pkg.reconciler.knativeeventing.reconciler.00-of-01\"","commit":"7ef32df-dirty","knative.dev/pod":"knative-operator-746cb8575-l7gpv"}
{"severity":"ERROR","timestamp":"2023-08-17T00:47:42.64058547Z","logger":"knative-operator","caller":"knativeserving/reconciler.go:295","message":"Returned an error","commit":"7ef32df-dirty","knative.dev/pod":"knative-operator-746cb8575-l7gpv","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"91dc27b9-bd12-49e2-ac09-0586633d5728","knative.dev/key":"knative-serving/knative-serving","targetMethod":"ReconcileKind","error":"the manifests of the target version 1.9.2 are not available to this release","stacktrace":"knative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving.(*reconcilerImpl).Reconcile\n\tknative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving/reconciler.go:295\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20230718152110-aef227e72ead/controller/controller.go:542\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20230718152110-aef227e72ead/controller/controller.go:491"}
{"severity":"ERROR","timestamp":"2023-08-17T00:47:42.640681287Z","logger":"knative-operator","caller":"controller/controller.go:566","message":"Reconcile error","commit":"7ef32df-dirty","knative.dev/pod":"knative-operator-746cb8575-l7gpv","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"91dc27b9-bd12-49e2-ac09-0586633d5728","knative.dev/key":"knative-serving/knative-serving","duration":"30.093542733s","error":"the manifests of the target version 1.9.2 are not available to this release","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20230718152110-aef227e72ead/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20230718152110-aef227e72ead/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20230718152110-aef227e72ead/controller/controller.go:491"}

CRD

---
# Source: knative/templates/knative-serving.yaml
apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  version: 1.9.2
  additionalManifests:
    - URL: https://github.com/knative/net-certmanager/releases/download/knative-v1.9.2/release.yaml
  podDisruptionBudgets:
    - name: "activator-pdb"
      minAvailable: "0%"
    - name: "webhook-pdb"
      minAvailable: "0%"
  ingress:
    istio:
      enabled: false
    kourier:
      enabled: true
  config:
    features:
      "kubernetes.podspec-volumes-emptydir": "enabled"
      "kubernetes.podspec-fieldref": "enabled"
      "kubernetes.podspec-securitycontext": "enabled"
      "queueproxy.mount-podinfo": "enabled"
    certmanager:
      issuerRef: |
        kind: ClusterIssuer
        name: zerossl
    network:
      ingress-class: "kourier.ingress.networking.knative.dev"
      namespace-wildcard-cert-selector: '{"matchExpressions": [{"key":"networking.knative.dev/disableWildcardCert","operator": "NotIn", "values":["true"]}]}'
      auto-tls: "Enabled"
      http-protocol: "Redirected"
    domain:
      "our-domain": ""
    observability:
      "logging.enable-request-log": "true"
      "logging.enable-probe-request-log": "false"
  services:
    - name: kourier
      annotations:
        "external-dns.alpha.kubernetes.io/ttl": "30"
        "external-dns.alpha.kubernetes.io/hostname": "our-domain"
        "service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags": "cluster=dlvr-test"

knative / operator

the manifests of the target version 1.9.2 are not available to this release #1533