Applying tanzu-addons-manager PackageInstall on existing cluster fails

[vijaykatam]

Bug description Applying PackageInstall such as

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: addons-manager.tanzu.vmware.com.2.0.0
  namespace: tkg-system
spec:
  refName: addons-manager.tanzu.vmware.com
  version: 2.0.0
  licenses:
    - "UNKNOWN"
  template:
    spec:
      fetch:
        - imgpkgBundle:
            image: index.docker.io/vkat/addons-manager-pkg@sha256:5b6dc6917876ccd81c6e5fde6e116770e5908083bac131b485d2bd225b64d0cd
      template:
        - ytt:
            paths:
              - config/
        - kbld:
            paths:
              - "-"
              - .imgpkg/images.yml
      deploy:
        - kapp: {}
---
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: tanzu-addons-manager
  namespace: tkg-system
spec:
  packageRef:
    refName: addons-manager.tanzu.vmware.com
    versionSelection:
      constraints: 2.0.0
  serviceAccountName: tkg-addons-app-sa
  syncPeriod: 5m0s
  values:
    - secretRef:
        name: tanzu-addons-manager-data-values
---
apiVersion: v1
kind: Secret
metadata:
  name: tanzu-addons-manager-data-values
  namespace: tkg-system
type: Opaque
stringData:
  values.yaml: |
    tanzuAddonsManager:
      namespace: tkg-system
      createNamespace: false
      deployment:
        hostNetwork: true
        priorityClassName: system-cluster-critical
        tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
      featureGates:
        clusterBootstrapController: true

fails with

apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  annotations:
    ext.packaging.carvel.dev/ytt-data-values-overlays: ""
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"packaging.carvel.dev/v1alpha1","kind":"PackageInstall","metadata":{"annotations":{},"name":"tanzu-addons-manager","namespace":"tkg-system"},"spec":{"packageRef":{"refName":"addons-manager.tanzu.vmware.com","versionSelection":{"constraints":"2.0.0"}},"serviceAccountName":"tkg-addons-app-sa","syncPeriod":"5m0s","values":[{"secretRef":{"name":"tanzu-addons-manager-data-values"}}]}}
    tkg.tanzu.vmware.com/addon-name: cluster-class-0.18-tanzu-addons-manager-addon
    tkg.tanzu.vmware.com/addon-namespace: tkg-system
    tkg.tanzu.vmware.com/addon-type: addons-management/tanzu-addons-manager
  creationTimestamp: "2022-03-18T17:30:43Z"
  finalizers:
  - finalizers.packageinstall.packaging.carvel.dev/delete
  generation: 3
  name: tanzu-addons-manager
  namespace: tkg-system
  resourceVersion: "36346"
  uid: 3dd3f078-310a-47ec-888c-259932e64db4
spec:
  packageRef:
    refName: addons-manager.tanzu.vmware.com
    versionSelection:
      constraints: 2.0.0
      prereleases: {}
  serviceAccountName: tkg-addons-app-sa
  syncPeriod: 5m0s
  values:
  - secretRef:
      name: tanzu-addons-manager-data-values
status:
  conditions:
  - message: Error (see .status.usefulErrorMessage for details)
    status: "True"
    type: ReconcileFailed
  friendlyDescription: 'Reconcile failed: Error (see .status.usefulErrorMessage for
    details)'
  lastAttemptedVersion: 2.0.0
  observedGeneration: 3
  usefulErrorMessage: |-
    kapp: Error: Validation errors:
    - Expected 'kind' on resource '/ () cluster' to be non-empty (stdin doc 6)
    - Expected 'apiVersion' on resource '/ () cluster' to be non-empty (stdin doc 6)
    - Expected 'metadata.name' on resource '/ () cluster' to be non-empty (stdin doc 6)
  version: 2.0.0

Affected product area (please put an X in all that apply)

• [ ] APIs
• [x] Addons
• [ ] CLI
• [ ] Docs
• [ ] IAM
• [ ] Installation
• [ ] Plugin
• [ ] Security
• [ ] Test and Release
• [ ] User Experience

Expected behavior PackageInstall succeeds without deleting and recreating

[vijaykatam]

@shyaamsn - Could you triage to assign to someone to fix the new addons-manager package?