Setting custom `spec.application.route.host` not working using the RHDH Operator on OpenShift, due to inconsistent permissions between downstream and upstream CSVs

/kind bug

What versions of software are you using?

RHDH operator next version (image: registry.redhat.io/rhdh/rhdh-rhel9-operator@sha256:f9e4e29b935cae26df62191e59f8240cddcc160d0ed29efe9d7d6f9ac549bc8e)
or RHDH operator latest released version (image: registry.redhat.io/rhdh/rhdh-rhel9-operator@sha256:9bea7eabdea44342248dfd6091a8f5c3b6e65884c916b3c2a073a4a64481aa6f)
OpenShift 4.15.0

What did you do exactly?

Against an OpenShift cluster:

# Similar issue when installing the released operator from the OpenShift OperatorHub
.rhdh/scripts/install-rhdh-catalog-source.sh --next --install-operator rhdh

cat <<EOF | oc apply -f -
apiVersion: rhdh.redhat.com/v1alpha1
kind: Backstage
metadata:
  name: test-bs-route
spec:
  application:
    route:
      host: test-bs-route.example.com
EOF

Actual behavior

The CR status is DeployFailed, and not all resources are created.

Using the upcoming 1.2 (--next)

The error message is failed to patch object *v1.Deployment: deployments.apps "backstage-test-bs-route" is forbidden: User "system:serviceaccount:rhdh-operator:rhdh-operator" cannot patch resource "deployments" in API group "apps" in the namespace "my-ns", which indicates a missing role for the operator service account.

Using the released 1.1.2 (--latest)

The error message is failed to deploy Backstage Route: Route.route.openshift.io "backstage-test-bs-route" is invalid: spec.host: Forbidden: you do not have permission to set the host field of the route, which also indicates a missing role for the operator service account.

Expected behavior

The downstream RHDH operator should reconcile successfully without any errors, and the Route created by the Operator should have the specified host set.

It works as expected when running the Backstage operator (using make deploy or make run), not the downstream RHDH bundle. This indicates an issue with out-of-sync RBAC permissions between both bundles, as already caught in https://github.com/janus-idp/operator/pull/351#discussion_r1599790206

We should make sure to keep those permissions in sync.

Any logs, error output, etc?

Using the upcoming 1.2 (--next):

$ oc get route
No resources found in my-ns namespace.

$ oc get statefulset                           
No resources found in my-ns namespace.

$ oc get deployment                                                                                                                           
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
backstage-test-bs-route   0/1     1            0           14m

$ oc describe test-bs-route

Name:         test-bs-route                                                                                                                                                                    
Namespace:    my-ns                                                                                                                                                                            
Labels:       <none>                                                                                                                                                                           
Annotations:  <none>                                                                                                                                                                           
API Version:  rhdh.redhat.com/v1alpha1                                                                                                                                                         
Kind:         Backstage                                                                                                                                                                        
Metadata:                                                                                                                                                                                      
  Creation Timestamp:  2024-05-14T12:34:22Z                                                                                                                                                    
  Generation:          1                                                                                                                                                                       
  Resource Version:    221740                                                                                                                                                                  
  UID:                 afaae393-0eeb-48a2-bbe7-32b1b224c856                                                                                                                                    
Spec:                                                                                                                                                                                          
  Application:                                                                                                                                                                                 
    Replicas:  1                                                                                                                                                                               
    Route:                                                                                                                                                                                     
      Enabled:  true                                                                                                                                                                           
      Host:     test-bs-route.example.com                                                                                                                                                      
Status:                                                                                                                                                                                        
  Conditions: 
    Message:               failed to apply backstage objects failed to patch object &Deployment{ObjectMeta:{backstage-test-bs-route  my-ns   218838 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[app.kubernetes.io/instance:test-bs-route app.kubernetes.io/name:backstage] map[deployment.kubernetes.io/revision:1] [{rhdh.redhat.com/v1alpha1 Backstage test-bs-route afaae393-0eeb-48a2-bbe7-32b1b224c856 0xc00080bd19 0xc00080bd18}] [] []},Spec:DeploymentSpec{Replicas:*1,Selector:&v1.LabelSelector{MatchLabels:map[string]string{rhdh.redhat.com/app: backstage-test-bs-route,},MatchExpressions:[]LabelSelectorRequirement{},},Template:{{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[rhdh.redhat.com/app:backstage-test-bs-route] map[] [] [] []} {[{dynamic-plugins-root {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil EphemeralVolumeSource{VolumeClaimTemplate:&PersistentVolumeClaimTemplate{ObjectMeta:{      0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:PersistentVolumeClaimSpec{AccessModes:[ReadWriteOnce],Resources:VolumeResourceRequirements{Limits:ResourceList{},Requests:ResourceList{storage: {{2147483648 0} {<nil>} 2Gi BinarySI},},},VolumeName:,Selector:nil,StorageClassName:nil,VolumeMode:nil,DataSource:nil,DataSourceRef:nil,VolumeAttributesClassName:nil,},},}}} {dynamic-plugins-npmrc {nil nil nil nil nil &SecretVolumeSource{SecretName:dynamic-plugins-npmrc,Items:[]KeyToPath{},DefaultMode:*420,Optional:*true,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {backstage-appconfig-test-bs-route {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:backstage-appconfig-test-bs-route,},Items:[]KeyToPath{},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}} {backstage-dynamic-plugins-test-bs-route {nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil &ConfigMapVolumeSource{LocalObjectReference:LocalObjectReference{Name:backstage-dynamic-plugins-test-bs-route,},Items:[]KeyToPath{},DefaultMode:*420,Optional:*false,} nil nil nil nil nil nil nil nil nil nil}}] [{install-dynamic-plugins registry.redhat.io/rhdh/rhdh-hub-rhel9@sha256:793df937f0739f0a2d328883f53c65fae57a3b6a060dff603e07f2c751b90e7b [./install-dynamic-plugins.sh /dynamic-plugins-root] [] /opt/app-root/src [] [] [{NPM_CONFIG_USERCONFIG /opt/app-root/src/.npmrc.dynamic-plugins nil}] {map[cpu:{{1000 -3} {<nil>}  DecimalSI} ephemeral-storage:{{5368709120 0} {<nil>} 5Gi BinarySI} memory:{{0 0} {0xc0006944e0}  BinarySI}] map[cpu:{{250 -3} {<nil>} 250m DecimalSI} memory:{{268435456 0} {<nil>}  BinarySI}] []} [] <nil> [{dynamic-plugins-root false /dynamic-plugins-root  <nil> } {dynamic-plugins-npmrc true /opt/app-root/src/.npmrc.dynamic-plugins .npmrc <nil> } {backstage-dynamic-plugins-test-bs-route true /opt/app-root/src/dynamic-plugins.yaml dynamic-plugins.yaml <nil> }] [] nil nil nil nil   IfNotPresent &SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} false false false}] [{backstage-backend registry.redhat.io/rhdh/rhdh-hub-rhel9@sha256:793df937f0739f0a2d328883f53c65fae57a3b6a060dff603e07f2c751b90e7b [] [--config dynamic-plugins-root/app-config.dynamic-plugins.yaml --config /opt/app-root/src/default.app-config.yaml]  [{backend 0 7007  }] [{ nil &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:backstage-envs-test-bs-route,},Optional:nil,}} { nil &SecretEnvSource{LocalObjectReference:LocalObjectReference{Name:backstage-db-test-bs-route,},Optional:nil,}}] [{APP_CONFIG_backend_listen_port 7007 nil}] {map[cpu:{{1000 -3} {<nil>}  DecimalSI} ephemeral-storage:{{5368709120 0} {<nil>} 5Gi BinarySI} memory:{{0 0} {0xc000694330}  BinarySI}] map[cpu:{{250 -3} {<nil>} 250m DecimalSI} memory:{{268435456 0} {<nil>}  BinarySI}] []} [] <nil> [{dynamic-plugins-root false /opt/app-root/src/dynamic-plugins-root  <nil> } {backstage-appconfig-test-bs-route true /opt/app-root/src/default.app-config.yaml default.app-config.yaml <nil> }] [] &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthcheck,Port:{0 7007 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:60,TimeoutSeconds:2,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} &Probe{ProbeHandler:ProbeHandler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthcheck,Port:{0 7007 },Host:,Scheme:HTTP,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,GRPC:nil,},InitialDelaySeconds:30,TimeoutSeconds:2,PeriodSeconds:10,SuccessThreshold:2,FailureThreshold:3,TerminationGracePeriodSeconds:nil,} nil nil   IfNotPresent &SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:*true,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:*false,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,} false false false}] []  <nil> <nil>  map[]   0xc00080bac0  false false false <nil> nil []   nil  [] []  <nil> nil [] <nil> <nil> <nil> map[] [] <nil> nil <nil> [] []}},Strategy:DeploymentStrategy{Type:,RollingUpdate:nil,},MinReadySeconds:0,RevisionHistoryLimit:nil,Paused:false,ProgressDeadlineSeconds:nil,},Status:DeploymentStatus{ObservedGeneration:0,Replicas:0,UpdatedReplicas:0,AvailableReplicas:0,UnavailableReplicas:0,Conditions:[]DeploymentCondition{},ReadyReplicas:0,CollisionCount:nil,},}: failed to patch object *v1.Deployment: deployments.apps "backstage-test-bs-route" is forbidden: User "system:serviceaccount:rhdh-operator:rhdh-operator" cannot patch resource "deployments" in API group "apps" in the namespace "my-ns"
    Reason:                DeployFailed
    Status:                False
    Type:                  Deployed
Events:                    <none>

Using the released 1.1.2 (--latest):

$ oc get route
No resources found in my-ns namespace.

$ oc get statefulset 
NAME                      READY   AGE
backstage-psql-test-bs-route   1/1     4m9s

$ oc get statefulset 
NAME                      READY   AGE
backstage-psql-test-bs-route   1/1     4m9s

$ oc describe backstage test-bs-route
Name:         test-bs-route
Namespace:    my-ns
Labels:       <none>
Annotations:  <none>
API Version:  rhdh.redhat.com/v1alpha1
Kind:         Backstage
Metadata:
  Creation Timestamp:  2024-05-15T07:20:33Z
  Generation:          1
  Resource Version:    316845
  UID:                 da912858-8314-4c1d-a176-a9bc63715745
Spec:
  Application:
    Replicas:  1
    Route:
      Enabled:  true
      Host:     test-bs-route.example.com
Status:
  Conditions:
    Last Transition Time:  2024-05-15T07:20:33Z
    Message:               failed to deploy Backstage Route: Route.route.openshift.io "backstage-test-bs-route" is invalid: spec.host: Forbidden: you do not have permission to set the host field of the route
    Reason:                DeployFailed
    Status:                False
    Type:                  Deployed
Events:                    <none>

janus-idp / operator