RHACM 2.9 subscription channel no longer available in `redhat-operators` catalog

[nastacio]

Describe the bug

I tried to apply the recent changes made to the repo in feature #298 on a new cluster using OCP 4.12 and then on a new cluster using OCP 4.13.

Deployments worked on Wednesday (11/22) without any problems. However, starting a couple days later, either on Friday or Saturday (11/24 or 11/25,) the subscription on both clusters generated an alert about the subscription channel release-2.9 no longer being available in the redhat-operators catalog source (see below.)

On a related note, and maybe that is just a coincidence, Red Hat container registry suffered an outage on 11/25 and I wonder if the reinstatement of some images may have restored an older or corrupted version of the catalog source.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: advanced-cluster-management
  namespace: open-cluster-management
spec:
  channel: release-2.9
  installPlanApproval: Automatic
  name: advanced-cluster-management
  source: redhat-operators
  sourceNamespace: openshift-marketplace
status:
  catalogHealth:
   ...
    - catalogSourceRef:
        apiVersion: operators.coreos.com/v1alpha1
        kind: CatalogSource
        name: redhat-operators
        namespace: openshift-marketplace
      healthy: true
  conditions:
    - lastTransitionTime: '2023-11-26T14:11:26Z'
      message: all available catalogsources are healthy
      reason: AllCatalogSourcesHealthy
      status: 'False'
      type: CatalogSourcesUnhealthy
    - message: >-
        constraints not satisfiable: no operators found in channel release-2.9
        of package advanced-cluster-management in the catalog referenced by
        subscription advanced-cluster-management, subscription
        advanced-cluster-management exists
      reason: ConstraintsNotSatisfiable
      status: 'True'
      type: ResolutionFailed

To Reproduce Steps to reproduce the behavior: In order to simplify the troubleshooting, you can validate this issue directly in the OCP console, based on the RHACM 2.9 instructions

1. Apply the Subscription YAML below to any OCP cluster on versions 4.12 or 4.13 (I have not tried it with 4.14 yet)
2. Wait for the status.conditions field to report the error about the constraints not satisfiable

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: advanced-cluster-management
  namespace: open-cluster-management
spec:
  channel: release-2.9
  installPlanApproval: Automatic
  name: advanced-cluster-management
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Expected behavior The Subscription should work and the operator should deploy without issue.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context On a 4.13 cluster:

[nastacio]

Looks like the catalog is being rebuilt. I just saw release-2.8 showing up again (still no release-2.9)

[nastacio]

The subscription channel is back in the Red Hat catalog and I am able to deploy RHACM 2.9 following this repo again.