Hello, we've discovered some inconsistent behavior from the rbac-backend plugin when scaling the backstage deployment to 2 pods/containers with identical instance. The use case we had issue with is when using the API endpoint PUT </api/permission/roles/:kind/:namespace/:name> (https://github.com/janus-idp/backstage-plugins/blob/main/plugins/rbac-backend/docs/apis.md#put-role) to update membership of a role. When having two Kubernetes pods running, the API request was received and processed by pod A, in which itself also updated the DB with the new role definition. Everything seemed good at this point until someone started querying the endpoint GET </api/permission/roles/:kind/:namespace/:name> (https://github.com/janus-idp/backstage-plugins/blob/main/plugins/rbac-backend/docs/apis.md#get-role) to retrieve the role - the updated role data was returned when a request hit pod A, however if the same request was received by pod B, we'd still get the old role definition.
Since the DB appears to be successfully updated, we are wondering if the plugin utilizes some kind of in memory cache that caused this out-of-sync behavior. Our backstage deployment has been configured to using Redis as the cache backend and it's seemingly not taking advantage of that. Does anyone have any thoughts on this? Thanks in advance.
Expected Behavior
When having a high availability deployment setup, API endpoints served by the RBAC backend plugin should provide consistent data across all instances.
What are the steps to reproduce this bug?
Horizontally scale the backstage deployment to have 2 replicas
Choose any existing RBAC role, and retrieve its current definition with the endpoint GET </api/permission/roles/:kind/:namespace/:name>
Modify the retrieved definition by adding a new member to the role. Feed the endpoint PUT </api/permission/roles/:kind/:namespace/:name> with the new definition to update the role
Querying GET </api/permission/roles/:kind/:namespace/:name>, you will see that the data returned varies depending on the pod/container being hit
crazy-canux
commented
3 months ago
Describe the bug
Hello, we've discovered some inconsistent behavior from the rbac-backend plugin when scaling the backstage deployment to 2 pods/containers with identical instance. The use case we had issue with is when using the API endpoint
PUT </api/permission/roles/:kind/:namespace/:name>(https://github.com/janus-idp/backstage-plugins/blob/main/plugins/rbac-backend/docs/apis.md#put-role) to update membership of a role. When having two Kubernetes pods running, the API request was received and processed by pod A, in which itself also updated the DB with the new role definition. Everything seemed good at this point until someone started querying the endpointGET </api/permission/roles/:kind/:namespace/:name>(https://github.com/janus-idp/backstage-plugins/blob/main/plugins/rbac-backend/docs/apis.md#get-role) to retrieve the role - the updated role data was returned when a request hit pod A, however if the same request was received by pod B, we'd still get the old role definition.Since the DB appears to be successfully updated, we are wondering if the plugin utilizes some kind of in memory cache that caused this out-of-sync behavior. Our backstage deployment has been configured to using Redis as the cache backend and it's seemingly not taking advantage of that. Does anyone have any thoughts on this? Thanks in advance.
Expected Behavior
When having a high availability deployment setup, API endpoints served by the RBAC backend plugin should provide consistent data across all instances.
What are the steps to reproduce this bug?
GET </api/permission/roles/:kind/:namespace/:name>PUT </api/permission/roles/:kind/:namespace/:name>with the new definition to update the roleGET </api/permission/roles/:kind/:namespace/:name>, you will see that the data returned varies depending on the pod/container being hitVersions of software used and environment
@backstage/app-defaults 1.5.3 @backstage/backend-app-api 0.5.14, 0.6.2 @backstage/backend-common 0.19.9, 0.20.1, 0.21.6 @backstage/backend-dev-utils 0.1.4 @backstage/backend-openapi-utils 0.1.9 @backstage/backend-plugin-api 0.6.16 @backstage/backend-tasks 0.5.21 @backstage/backend-test-utils 0.3.6 @backstage/catalog-client 1.6.3 @backstage/catalog-model 1.4.5 @backstage/cli 0.26.2 @backstage/cli-common 0.1.13 @backstage/cli-node 0.2.4 @backstage/config 1.2.0 @backstage/config-loader 1.7.0 @backstage/core-app-api 1.12.3 @backstage/core-compat-api 0.2.3 @backstage/core-components 0.12.5, 0.13.10, 0.14.3 @backstage/core-plugin-api 1.9.1 @backstage/dev-utils 1.0.30 @backstage/e2e-test-utils 0.1.1 @backstage/errors 1.2.4 @backstage/eslint-plugin 0.1.6 @backstage/frontend-plugin-api 0.6.3 @backstage/integration 1.9.1 @backstage/integration-aws-node 0.1.12 @backstage/integration-react 1.1.25 @backstage/plugin-adr 0.6.16 @backstage/plugin-adr-backend 0.4.13 @backstage/plugin-adr-common 0.2.21 @backstage/plugin-api-docs 0.11.3 @backstage/plugin-app-backend 0.3.64 @backstage/plugin-app-node 0.1.16 @backstage/plugin-auth-backend 0.22.3 @backstage/plugin-auth-backend-module-atlassian-provider 0.1.8 @backstage/plugin-auth-backend-module-aws-alb-provider 0.1.8 @backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.11 @backstage/plugin-auth-backend-module-github-provider 0.1.13 @backstage/plugin-auth-backend-module-gitlab-provider 0.1.13 @backstage/plugin-auth-backend-module-google-provider 0.1.13 @backstage/plugin-auth-backend-module-microsoft-provider 0.1.11 @backstage/plugin-auth-backend-module-oauth2-provider 0.1.13 @backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.9 @backstage/plugin-auth-backend-module-oidc-provider 0.1.7 @backstage/plugin-auth-backend-module-okta-provider 0.0.9 @backstage/plugin-auth-node 0.4.11 @backstage/plugin-auth-react 0.0.3 @backstage/plugin-catalog 1.18.2 @backstage/plugin-catalog-backend 1.21.0 @backstage/plugin-catalog-backend-module-aws 0.3.11 @backstage/plugin-catalog-backend-module-gitlab 0.3.14 @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.14 @backstage/plugin-catalog-backend-module-unprocessed 0.4.3 @backstage/plugin-catalog-common 1.0.22 @backstage/plugin-catalog-graph 0.4.3 @backstage/plugin-catalog-import 0.10.9 @backstage/plugin-catalog-node 1.11.0 @backstage/plugin-catalog-react 1.11.2 @backstage/plugin-catalog-unprocessed-entities 0.2.2 @backstage/plugin-catalog-unprocessed-entities-common 0.0.1 @backstage/plugin-devtools 0.1.12 @backstage/plugin-devtools-backend 0.3.2 @backstage/plugin-devtools-common 0.1.9 @backstage/plugin-events-node 0.2.21, 0.3.2 @backstage/plugin-github-actions 0.6.14 @backstage/plugin-home 0.7.2 @backstage/plugin-home-react 0.1.11 @backstage/plugin-kubernetes 0.11.8 @backstage/plugin-kubernetes-backend 0.16.4 @backstage/plugin-kubernetes-common 0.7.5 @backstage/plugin-kubernetes-node 0.1.10 @backstage/plugin-kubernetes-react 0.3.3 @backstage/plugin-org 0.6.23 @backstage/plugin-pagerduty 0.7.5 @backstage/plugin-permission-backend 0.5.40 @backstage/plugin-permission-common 0.7.13 @backstage/plugin-permission-node 0.7.27 @backstage/plugin-permission-react 0.4.21 @backstage/plugin-proxy-backend 0.4.14 @backstage/plugin-scaffolder 1.19.2 @backstage/plugin-scaffolder-backend 1.22.3 @backstage/plugin-scaffolder-backend-module-azure 0.1.8 @backstage/plugin-scaffolder-backend-module-bitbucket 0.2.6 @backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.6 @backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.6 @backstage/plugin-scaffolder-backend-module-cookiecutter 0.2.40 @backstage/plugin-scaffolder-backend-module-gerrit 0.1.8 @backstage/plugin-scaffolder-backend-module-gitea 0.1.6 @backstage/plugin-scaffolder-backend-module-github 0.2.6 @backstage/plugin-scaffolder-backend-module-gitlab 0.3.2 @backstage/plugin-scaffolder-common 1.5.1 @backstage/plugin-scaffolder-node 0.2.9, 0.4.2 @backstage/plugin-scaffolder-react 1.8.3 @backstage/plugin-search 1.4.9 @backstage/plugin-search-backend 1.5.6 @backstage/plugin-search-backend-module-catalog 0.1.21 @backstage/plugin-search-backend-module-elasticsearch 1.3.19 @backstage/plugin-search-backend-module-pg 0.5.25 @backstage/plugin-search-backend-module-techdocs 0.1.21 @backstage/plugin-search-backend-node 1.2.20 @backstage/plugin-search-common 1.2.11 @backstage/plugin-search-react 1.7.9 @backstage/plugin-shortcuts 0.3.22 @backstage/plugin-signals-node 0.0.1 @backstage/plugin-signals-react 0.0.1 @backstage/plugin-sonarqube 0.7.15 @backstage/plugin-sonarqube-backend 0.2.18 @backstage/plugin-sonarqube-react 0.1.14 @backstage/plugin-tech-radar 0.7.2 @backstage/plugin-techdocs 1.10.3 @backstage/plugin-techdocs-backend 1.10.3 @backstage/plugin-techdocs-module-addons-contrib 1.1.8 @backstage/plugin-techdocs-node 1.12.2 @backstage/plugin-techdocs-react 1.2.2 @backstage/plugin-user-settings 0.8.4 @backstage/release-manifests 0.0.11 @backstage/test-utils 1.5.3 @backstage/theme 0.2.19, 0.4.4, 0.5.2 @backstage/types 1.1.1 @backstage/version-bridge 1.0.7 @janus-idp/backstage-plugin-feedback 1.0.1 @janus-idp/backstage-plugin-feedback-backend 1.1.1 @janus-idp/backstage-plugin-rbac 1.13.2 @janus-idp/backstage-plugin-rbac-backend 2.5.0 @janus-idp/backstage-plugin-rbac-common 1.2.1, 1.3.1 @janus-idp/backstage-plugin-rbac-node 1.0.4 @k-phoen/backstage-plugin-confluence 0.0.15 @k-phoen/backstage-plugin-confluence-backend 0.0.14 @k-phoen/backstage-plugin-grafana 0.1.22 @roadiehq/backstage-plugin-argo-cd 2.5.1 @roadiehq/backstage-plugin-datadog 2.2.2 @roadiehq/backstage-plugin-home-rss 1.3.5