knative-serving is not ready. "Install failed with message: invalid JSON document"

[johnnykim]

Describe the bug knative-serving is not getting ready.

$ kubectl get -n knative-serving KnativeServing
NAME              VERSION   READY   REASON
knative-serving   1.3.1     False   Error

$ kubectl get -n knative-serving KnativeServing -o yaml
apiVersion: v1
items:
- apiVersion: operator.knative.dev/v1beta1
  kind: KnativeServing
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"operator.knative.dev/v1beta1","kind":"KnativeServing","metadata":{"annotations":{},"name":"knative-serving","namespace":"knative-serving"},"spec":{"config":{"domain":{"ai.heydealer.com":""}},"version":"1.3"}}
    creationTimestamp: "2022-06-23T07:43:47Z"
    finalizers:
    - knativeservings.operator.knative.dev
    generation: 23
    name: knative-serving
    namespace: knative-serving
    resourceVersion: "1751311742"
    uid: c56ce34f-5f64-49ef-bf40-071fdb90480f
  spec:
    config:
      domain:
        ai.heydealer.com: ""
    version: "1.3"
  status:
    conditions:
    - lastTransitionTime: "2022-06-23T07:44:05Z"
      status: "True"
      type: DependenciesInstalled
    - lastTransitionTime: "2023-11-01T03:40:22Z"
      status: "True"
      type: DeploymentsAvailable
    - lastTransitionTime: "2023-11-07T02:47:08Z"
      message: 'Install failed with message: invalid JSON document'
      reason: Error
      status: "False"
      type: InstallSucceeded
    - lastTransitionTime: "2023-11-07T02:47:08Z"
      message: 'Install failed with message: invalid JSON document'
      reason: Error
      status: "False"
      type: Ready
    - lastTransitionTime: "2023-11-02T08:04:40Z"
      status: "True"
      type: VersionMigrationEligible
    manifests:
    - /var/run/ko/knative-serving/1.3.1
    observedGeneration: 23
    version: 1.3.1
kind: List
metadata:
  resourceVersion: ""

• Installed Operator

  • https://github.com/knative/operator/releases/download/knative-v1.3.2/operator.yaml

• relative knative-operator logs

  {"severity":"INFO","timestamp":"2023-11-07T03:12:03.994800074Z","logger":"knative-operator","caller":"knativeserving/knativeserving.go:100","message":"Reconciling KnativeServing","knative.dev/pod":"knative-operator-66f7766b4f-brxfs","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"1ffb0962-6a47-482a-85ef-43bc56dd251c","knative.dev/key":"knative-serving/knative-serving","status":{"observedGeneration":23,"conditions":[{"type":"DependenciesInstalled","status":"True","lastTransitionTime":"2022-06-23T07:44:05Z"},{"type":"DeploymentsAvailable","status":"True","lastTransitionTime":"2023-11-01T03:40:22Z"},{"type":"InstallSucceeded","status":"False","lastTransitionTime":"2023-11-07T02:47:08Z","reason":"Error","message":"Install failed with message: invalid JSON document"},{"type":"Ready","status":"False","lastTransitionTime":"2023-11-07T02:47:08Z","reason":"Error","message":"Install failed with message: invalid JSON document"},{"type":"VersionMigrationEligible","status":"True","lastTransitionTime":"2023-11-02T08:04:40Z"}],"version":"1.3.1","manifests":["/var/run/ko/knative-serving/1.3.1"]}}
  {"severity":"ERROR","timestamp":"2023-11-07T03:12:07.714673304Z","logger":"knative-operator","caller":"knativeserving/reconciler.go:294","message":"Returned an error","knative.dev/pod":"knative-operator-66f7766b4f-brxfs","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"1ffb0962-6a47-482a-85ef-43bc56dd251c","knative.dev/key":"knative-serving/knative-serving","targetMethod":"ReconcileKind","error":"failed to apply non rbac manifest: invalid JSON document","stacktrace":"knative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving.(*reconcilerImpl).Reconcile\n\tknative.dev/operator/pkg/client/injection/reconciler/operator/v1beta1/knativeserving/reconciler.go:294\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220314170718-721abec0a377/controller/controller.go:535\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220314170718-721abec0a377/controller/controller.go:484"}
  {"severity":"ERROR","timestamp":"2023-11-07T03:12:07.714742577Z","logger":"knative-operator","caller":"controller/controller.go:559","message":"Reconcile error","knative.dev/pod":"knative-operator-66f7766b4f-brxfs","knative.dev/controller":"knative.dev.operator.pkg.reconciler.knativeserving.Reconciler","knative.dev/kind":"operator.knative.dev.KnativeServing","knative.dev/traceid":"1ffb0962-6a47-482a-85ef-43bc56dd251c","knative.dev/key":"knative-serving/knative-serving","duration":"3.720039923s","error":"failed to apply non rbac manifest: invalid JSON document","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20220314170718-721abec0a377/controller/controller.go:559\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20220314170718-721abec0a377/controller/controller.go:536\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20220314170718-721abec0a377/controller/controller.go:484"}
  {"severity":"INFO","timestamp":"2023-11-07T03:12:07.714873135Z","logger":"knative-operator.event-broadcaster","caller":"record/event.go:282","message":"Event(v1.ObjectReference{Kind:\"KnativeServing\", Namespace:\"knative-serving\", Name:\"knative-serving\", UID:\"c56ce34f-5f64-49ef-bf40-071fdb90480f\", APIVersion:\"operator.knative.dev/v1beta1\", ResourceVersion:\"1751311742\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' failed to apply non rbac manifest: invalid JSON document","knative.dev/pod":"knative-operator-66f7766b4f-brxfs"}

Knative release version

• knative-serving : 1.3.1

Additional context If I upgrade knative-serving 1.3 to 1.4 with ignoring this issue, kserve pod's queue-proxy container returns required key QUEUE_SERVING_TLS_PORT missing value error.

[houshengbo]

1.3? I am afraid this is not an active version any more.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.