keycloak-backend alpha plugin doesn't seem to be working

[airedwin]

Hi, I add this to my backstage deployment following the directions for adding it to the new backstage backend I'm still unable to login since it can't find my user identify in the catalog, i'm assuming this plugin automatically syncs, which I don't see happening during start up

Loading config from MergedConfigSource{FileConfigSource{path="/app/app-config-from-configmap.yaml"}, EnvConfigSource{count=4}} 55 {"level":"info","message":"Found 1 new secrets in config that will be redacted","service":"backstage"} 54 {"level":"info","message":"Listening on :7007","service":"rootHttpRouter"} 53 {"level":"info","message":"Plugin initialization started: 'app', 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'permission', 'search'","service":"backstage","type":"initialization"} 52 {"level":"info","message":"Serving static app content from /app/packages/app/dist","plugin":"app","service":"backstage"} 51 {"level":"info","message":"Starting scaffolder with the following actions enabled fetch:plain, fetch:plain:file, fetch:template, debug:log, debug:wait, catalog:register, catalog:fetch, catalog:write, fs:delete, fs:rename","plugin":"scaffolder","service":"backstage"} 50 {"level":"info","message":"Creating Local publisher for TechDocs","plugin":"techdocs","service":"backstage"} 49 {"level":"info","message":"Configuring \"database\" as KeyStore provider","plugin":"auth","service":"backstage"} 48 {"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage"} 47 {"level":"info","message":"Configuring auth provider: oauth2Proxy","plugin":"auth","service":"backstage"} 46 {"entry":"main","level":"info","message":"Injecting env config into module-backstage.686b5ec2.js","plugin":"app","service":"backstage"} 45 {"level":"info","message":"Added DefaultCatalogCollatorFactory collator factory for type software-catalog","plugin":"search","service":"backstage"} 44 {"level":"info","message":"Added DefaultTechDocsCollatorFactory collator factory for type techdocs","plugin":"search","service":"backstage"} 43 {"level":"warn","message":"Permission backend started with permissions disabled. Enable permissions by setting permission.enabled=true.","plugin":"permission","service":"backstage"} 42 {"level":"info","message":"Storing 291 updated assets and 0 new assets","plugin":"app","service":"backstage"} 41 {"level":"info","message":"Starting all scheduled search tasks.","plugin":"search","service":"backstage"} 40 {"level":"info","message":"Plugin initialization complete, newly initialized: 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'search', 'permission', 'app'","service":"backstage","type":"initialization"} 39 {"level":"info","message":"Task worker starting: search_index_software_catalog, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_software_catalog"} 38 {"level":"info","message":"Task worker startin

[rm3l]

Hi @airedwin ,

Thanks for reporting this issue. Can you please clarify:

• version of the application?
• version of the plugin?
• the exact reproduction steps, especially what you did add exactly and which directions you did follow, when you said "I add this to my backstage deployment following the directions for adding it to the new backstage backend"?
• the behavior you expected

That will help us better understand and scope this issue. Thanks.

[airedwin]

backstage version 1.27.0 using the alpha backend, this was defaulted through their create-app plugin 1.9.12

i followed the directions at https://janus-idp.io/plugins/keycloak/#new-backend-configuration my app-config.yaml looks like

catalog:
  providers:
    keycloakOrg:
      default:
        baseUrl: https://host/auth
        loginRealm: myrealm
        realm: myrealm
        clientId: client-confidential
        clientSecret: ${KEYCLOAK_CLIENTSECRET}
        schedule:
          frequency: { minutes: 30 }
          timeout: { minutes: 3 }
          initialDelay: { seconds: 15 }
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  rules:
    - allow: [Component, System, API, Resource, Location]
  locations:
    - type: file
      target: ../../examples/entities.yaml
    - type: file
      target: ../../examples/template/template.yaml
      rules:
        - allow: [Template]
    - type: file
      target: ../../examples/org.yaml
      rules:
        - allow: [User, Group]

i expect to see the scheduler running in the logs and for my keycloak users to be imported into the catalog

[airedwin]

any update on this? is there something that i am doing wrong?

[rm3l]

@Zaperex Can you please help answer this? Thanks.

[Zaperex]

@airedwin I'm going to need a bit more information.

• What version is your keycloak instance? If you're using keycloak v17+, you should omit the /auth endpoint in your baseUrl since that would usually result in a 404 Not Found error to be triggered when the keycloak-backend attempts to read from the keycloak instance.
• Also since you have set an initial delay of 15 seconds on your scheduler, it won't actually trigger immediately so the logs you provided would not have captured the setup of this. You would need to wait up to 15s before it starts the reading process. Can you provide more logs?
• If you are using keycloak 16 or earlier, then your app-config.yaml looks correct.
• Are you using a postgres database when starting backstage? If so, the schedule for the keycloak-backend would be stored in the database, and might not startup immediately after backend starts due to that next_update_at value in the database.
  • You can verify by going into the postgresql database, connecting to the catalog database with \c backstage_plugin_catalog, then querying: SELECT * FROM backstage_backend_tasks__tasks;
  • If an entry for the keycloak backend does not exist in that table, then it probably never got started. In that case, you should check if you have properly installed the keycloak backend into the backend.

I tested using your configuration of using:

• npx @backstage/create-app@0.5.15
• @janus-idp/backstage-plugin-keycloak-backend@1.9.12
• keycloak instance that is v22.0.3
• Also needed to add the following resolution into the root package.json when I tested due to the 1.28.x backstage packages being pulled in (due to transitive dependencies) preventing the backend from starting.

  "resolutions": {
  "@backstage/backend-app-api": "0.7.5",
  "@backstage/backend-dynamic-feature-service": "0.2.10"
  },

  Do you see a log containing the following that indicates the keycloak backend task worker started?

  2024-06-21T15:01:16.718Z catalog info Task worker starting: KeycloakOrgEntityProvider:default:refresh, {"version":2,"cadence":"PT30M","initialDelayDuration":"PT15S","timeoutAfterDuration":"PT3M"} task=KeycloakOrgEntityProvider:default:refresh

  A successful sync would log out something similar to:

  2024-06-21T15:01:31.722Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
  2024-06-21T15:01:31.778Z catalog info Read 5 Keycloak users and 4 Keycloak groups in 0.1 seconds. Committing... class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
  2024-06-21T15:01:31.783Z catalog info Committed 5 Keycloak users and 4 Keycloak groups in 0.0 seconds. class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6

  When I tested with /auth, I would get a 404 error:

  2024-06-21T15:02:43.423Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f
  2024-06-21T15:02:43.443Z catalog error Error while syncing Keycloak users and groups Request failed with status code 404 class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f name=Error stack=Error: Request failed with status code 404
  at createError (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/createError.js:16:15)
  at settle (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/settle.js:17:12)
  at IncomingMessage.handleStreamEnd (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/adapters/http.js:322:11)
  at IncomingMessage.emit (node:events:530:35)
  at endReadableNT (node:internal/streams/readable:1696:12)
  at process.processTicksAndRejections (node:internal/process/task_queues:82:21) status=404

  Do you have any additional logs related to the keycloak backend appear? Having those would help debug your issue.

Also can you check if you have your keycloak instance setup correctly? It should have the following

• client access type set to confidential
• service account roles are enabled
  • The following realm-management client roles are enabled:
  • query-groups
  • query-users
  • view-users

[airedwin]

@airedwin I'm going to need a bit more information.

• What version is your keycloak instance? If you're using keycloak v17+, you should omit the /auth endpoint in your baseUrl since that would usually result in a 404 Not Found error to be triggered when the keycloak-backend attempts to read from the keycloak instance.
• Also since you have set an initial delay of 15 seconds on your scheduler, it won't actually trigger immediately so the logs you provided would not have captured the setup of this. You would need to wait up to 15s before it starts the reading process. Can you provide more logs?
• If you are using keycloak 16 or earlier, then your app-config.yaml looks correct.

• Are you using a postgres database when starting backstage? If so, the schedule for the keycloak-backend would be stored in the database, and might not startup immediately after backend starts due to that next_update_at value in the database.

  • You can verify by going into the postgresql database, connecting to the catalog database with \c backstage_plugin_catalog, then querying: SELECT * FROM backstage_backend_tasks__tasks;
  • If an entry for the keycloak backend does not exist in that table, then it probably never got started. In that case, you should check if you have properly installed the keycloak backend into the backend.

I tested using your configuration of using:

• npx @backstage/create-app@0.5.15
• @janus-idp/backstage-plugin-keycloak-backend@1.9.12
• keycloak instance that is v22.0.3
• Also needed to add the following resolution into the root package.json when I tested due to the 1.28.x backstage packages being pulled in (due to transitive dependencies) preventing the backend from starting.

  "resolutions": {
    "@backstage/backend-app-api": "0.7.5",
    "@backstage/backend-dynamic-feature-service": "0.2.10"
  },

Do you see a log containing the following that indicates the keycloak backend task worker started?

2024-06-21T15:01:16.718Z catalog info Task worker starting: KeycloakOrgEntityProvider:default:refresh, {"version":2,"cadence":"PT30M","initialDelayDuration":"PT15S","timeoutAfterDuration":"PT3M"} task=KeycloakOrgEntityProvider:default:refresh

A successful sync would log out something similar to:

2024-06-21T15:01:31.722Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
2024-06-21T15:01:31.778Z catalog info Read 5 Keycloak users and 4 Keycloak groups in 0.1 seconds. Committing... class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6
2024-06-21T15:01:31.783Z catalog info Committed 5 Keycloak users and 4 Keycloak groups in 0.0 seconds. class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=94a7f8ba-4414-4173-bdfd-ae325c6217a6

When I tested with /auth, I would get a 404 error:

2024-06-21T15:02:43.423Z catalog info Reading Keycloak users and groups class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f
2024-06-21T15:02:43.443Z catalog error Error while syncing Keycloak users and groups Request failed with status code 404 class=KeycloakOrgEntityProvider taskId=KeycloakOrgEntityProvider:default:refresh taskInstanceId=ba54d185-7ff5-4086-b033-6c8f6ae9552f name=Error stack=Error: Request failed with status code 404
    at createError (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/home/frkong/coding/test-keycloak/node_modules/@keycloak/keycloak-admin-client/node_modules/axios/lib/adapters/http.js:322:11)
    at IncomingMessage.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) status=404

Do you have any additional logs related to the keycloak backend appear? Having those would help debug your issue.

Also can you check if you have your keycloak instance setup correctly? It should have the following

• client access type set to confidential

• service account roles are enabled

  • The following realm-management client roles are enabled:

  • query-groups

  • query-users

  • view-users

I will give this a try, i think the only difference between yours and mine is the resolutions? I dont see the Task starting in the logs, I will check the database shortly

[airedwin]

My start up logs are

Loading config from MergedConfigSource{FileConfigSource{path="/app/app-config-from-configmap.yaml"}, EnvConfigSource{count=4}}

{"level":"info","message":"Found 1 new secrets in config that will be redacted","service":"backstage"}

{"level":"info","message":"Listening on :7007","service":"rootHttpRouter"}

{"level":"info","message":"Plugin initialization started: 'app', 'proxy', 'scaffolder', 'techdocs', 'auth', 'catalog', 'permission', 'search'","service":"backstage","type":"initialization"}

{"level":"info","message":"Serving static app content from /app/packages/app/dist","plugin":"app","service":"backstage"}

{"level":"info","message":"Starting scaffolder with the following actions enabled fetch:plain, fetch:plain:file, fetch:template, debug:log, debug:wait, catalog:register, catalog:fetch, catalog:write, fs:delete, fs:rename","plugin":"scaffolder","service":"backstage"}

{"level":"info","message":"Creating Local publisher for TechDocs","plugin":"techdocs","service":"backstage"}

{"level":"info","message":"Configuring \"database\" as KeyStore provider","plugin":"auth","service":"backstage"}

{"level":"warn","message":"Permission backend started with permissions disabled. Enable permissions by setting permission.enabled=true.","plugin":"permission","service":"backstage"}

{"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage"}

{"level":"info","message":"Configuring auth provider: oauth2Proxy","plugin":"auth","service":"backstage"}

{"level":"info","message":"Added DefaultCatalogCollatorFactory collator factory for type software-catalog","plugin":"search","service":"backstage"}

{"level":"info","message":"Added DefaultTechDocsCollatorFactory collator factory for type techdocs","plugin":"search","service":"backstage"}

{"entry":"main","level":"info","message":"Injecting env config into module-backstage.686b5ec2.js","plugin":"app","service":"backstage"}

{"level":"info","message":"Storing 291 updated assets and 0 new assets","plugin":"app","service":"backstage"}

{"level":"info","message":"Starting all scheduled search tasks.","plugin":"search","service":"backstage"}

{"level":"info","message":"Plugin initialization complete, newly initialized: 'proxy', 'techdocs', 'scaffolder', 'permission', 'auth', 'search', 'catalog', 'app'","service":"backstage","type":"initialization"}

{"level":"info","message":"Task worker starting: search_index_software_catalog, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_software_catalog"}

{"level":"info","message":"Task worker starting: search_index_techdocs, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_techdocs"}

{"level":"info","message":"::1 - - [21/Jun/2024:17:02:53 +0000] \"GET /api/catalog/.backstage/auth/v1/jwks.json HTTP/1.1\" 200 11 \"-\" \"node\"","service":"rootHttpRouter","type":"incomingRequest"}

{"level":"info","message":"Created new signing key 2193e67b-262e-467d-bdd8-326760316a72","service":"backstage"}

{"level":"info","message":"::1 - - [21/Jun/2024:17:02:53 +0000] \"GET /api/search/.backstage/auth/v1/jwks.json HTTP/1.1\" 200 754 \"-\" \"node\"","service":"rootHttpRouter","type":"incomingRequest"}

{"level":"info","message":"::1 - - [21/Jun/2024:17:02:53 +0000] \"GET /api/search/.backstage/auth/v1/jwks.json HTTP/1.1\" 200 754 \"-\" \"jose/v5.4.0\"","service":"rootHttpRouter","type":"incomingRequest"}

{"level":"info","message":"::1 - - [21/Jun/2024:17:02:53 +0000] \"GET /api/search/.backstage/auth/v1/jwks.json HTTP/1.1\" 200 754 \"-\" \"jose/v5.4.0\"","service":"rootHttpRouter","type":"incomingRequest"}

{"level":"info","message":"::1 - - [21/Jun/2024:17:02:53 +0000] \"GET /api/catalog/entities?limit=500&filter=metadata.annotations.backstage.io%2Ftechdocs-ref&offset=0 HTTP/1.1\" 200 2 \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"rootHttpRouter","type":"incomingRequest"}

{"level":"info","message":"::1 - - [21/Jun/2024:17:02:53 +0000] \"GET /api/catalog/entities?limit=500&offset=0 HTTP/1.1\" 200 2 \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"rootHttpRouter","type":"incomingRequest"}

{"documentType":"techdocs","level":"warn","message":"Index for techdocs was not created: indexer received 0 documents","plugin":"search","service":"backstage"}

{"documentType":"techdocs","level":"info","message":"Collating documents for techdocs succeeded","plugin":"search","service":"backstage"}

{"documentType":"software-catalog","level":"warn","message":"Index for software-catalog was not created: indexer received 0 documents","plugin":"search","service":"backstage"}

{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog succeeded","plugin":"search","service":"backstage"}

[airedwin]

my packages/backend/src/index.ts is

/*
 * Hi!
 *
 * Note that this is an EXAMPLE Backstage backend. Please check the README.
 *
 * Happy hacking!
 */

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

backend.add(import('@backstage/plugin-app-backend/alpha'));
backend.add(import('@backstage/plugin-proxy-backend/alpha'));
backend.add(import('@backstage/plugin-scaffolder-backend/alpha'));
backend.add(import('@backstage/plugin-techdocs-backend/alpha'));

// auth plugin
backend.add(import('@backstage/plugin-auth-backend'));
// See https://backstage.io/docs/backend-system/building-backends/migrating#the-auth-plugin
backend.add(import('@backstage/plugin-auth-backend-module-guest-provider'));
// See https://backstage.io/docs/auth/guest/provider
backend.add(
  import('@backstage/plugin-auth-backend-module-oauth2-proxy-provider'),
);

// catalog plugin
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
backend.add(
  import('@backstage/plugin-catalog-backend-module-scaffolder-entity-model'),
);

// permission plugin
backend.add(import('@backstage/plugin-permission-backend/alpha'));
backend.add(
  import('@backstage/plugin-permission-backend-module-allow-all-policy'),
);

// search plugin
backend.add(import('@backstage/plugin-search-backend/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-catalog/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-techdocs/alpha'));

// custom keycloak plugin
// @ts-ignore
backend.add(import('@janus-idp/backstage-plugin-keycloak-backend/alpha'));

backend.start();

[airedwin]

I assume that is how I "install" the plugin?

[Zaperex]

your packages/backend/src/index.ts looks correct.

[airedwin]

It seems like I don't have those tables in my database, I'm using an AWS RDS instance, with a single database with multiple schemas

[Zaperex]

Hmm that most likely indicates that the keycloak backend did not start up at all.

[airedwin]

I built in docker with this dockerfile

# Stage 1 - Create yarn install skeleton layer
FROM node:18-bookworm-slim AS packages

WORKDIR /app
COPY package.json yarn.lock ./

COPY packages packages

# Comment this out if you don't have any internal plugins
#COPY plugins plugins

RUN find packages \! -name "package.json" -mindepth 2 -maxdepth 2 -exec rm -rf {} \+

# Stage 2 - Install dependencies and build packages
FROM node:18-bookworm-slim AS build

# Install isolate-vm dependencies, these are needed by the @backstage/plugin-scaffolder-backend.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends python3 g++ build-essential && \
    yarn config set python /usr/bin/python3

# Install sqlite3 dependencies. You can skip this if you don't use sqlite3 in the image,
# in which case you should also move better-sqlite3 to "devDependencies" in package.json.
#RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
#    --mount=type=cache,target=/var/lib/apt,sharing=locked \
#    apt-get update && \
#    apt-get install -y --no-install-recommends libsqlite3-dev

USER node
WORKDIR /app

COPY --from=packages --chown=node:node /app .

RUN yarn config set "strict-ssl" false -g
RUN --mount=type=cache,target=/home/node/.cache/yarn,sharing=locked,uid=1000,gid=1000 \
    yarn install --network-timeout 600000

COPY --chown=node:node . .

RUN yarn tsc
RUN yarn --cwd packages/backend build
# If you have not yet migrated to package roles, use the following command instead:
# RUN yarn --cwd packages/backend backstage-cli backend:bundle --build-dependencies

RUN mkdir packages/backend/dist/skeleton packages/backend/dist/bundle \
    && tar xzf packages/backend/dist/skeleton.tar.gz -C packages/backend/dist/skeleton \
    && tar xzf packages/backend/dist/bundle.tar.gz -C packages/backend/dist/bundle

# Stage 3 - Build the actual backend image and install production dependencies
FROM node:18-bookworm-slim

# Install isolate-vm dependencies, these are needed by the @backstage/plugin-scaffolder-backend.
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && \
    apt-get install -y --no-install-recommends python3 g++ build-essential && \
    yarn config set python /usr/bin/python3

# Install sqlite3 dependencies. You can skip this if you don't use sqlite3 in the image,
# in which case you should also move better-sqlite3 to "devDependencies" in package.json.
#RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
#    --mount=type=cache,target=/var/lib/apt,sharing=locked \
#    apt-get update && \
#    apt-get install -y --no-install-recommends libsqlite3-dev

# From here on we use the least-privileged `node` user to run the backend.
USER node

# This should create the app dir as `node`.
# If it is instead created as `root` then the `tar` command below will
# fail: `can't create directory 'packages/': Permission denied`.
# If this occurs, then ensure BuildKit is enabled (`DOCKER_BUILDKIT=1`)
# so the app dir is correctly created as `node`.
WORKDIR /app

# Copy the install dependencies from the build stage and context
COPY --from=build --chown=node:node /app/yarn.lock /app/package.json /app/packages/backend/dist/skeleton/ ./

RUN yarn config set "strict-ssl" false -g
RUN --mount=type=cache,target=/home/node/.cache/yarn,sharing=locked,uid=1000,gid=1000 \
    yarn install --production --network-timeout 600000

# Copy the built packages from the build stage
COPY --from=build --chown=node:node /app/packages/backend/dist/bundle/ ./

# Copy any other files that we need at runtime
COPY --chown=node:node app-config.yaml ./

# This switches many Node.js dependencies to production mode.
ENV NODE_ENV production

CMD ["node", "packages/backend", "--config", "app-config.yaml"]

[airedwin]

and i'm using a helm chart from https://github.com/backstage/charts

i slightly customized it to not deploy the in cluster postgresql container, but instead to use environment values for the RDS

[Zaperex]

I see, can I ask if you had the same issue when running it locally?

[Zaperex]

sorry did not mean to close it

[airedwin]

i have not tried running it locally

[Zaperex]

Can you try running it locally to verify that it doesn't work there as well? It should also make this debugging process faster so we can see where the issue stems from.

[airedwin]

ok let me set up a local db

[airedwin]

interesting, it worked locally with same docker image

{"level":"info","message":"Task worker starting: KeycloakOrgEntityProvider:default:refresh, {\"version\":2,\"cadence\":\"PT30M\",\"initialDelayDuration\":\"PT15S\",\"timeoutAfterDuration\":\"PT3M\"}","plugin":"catalog","service":"backstage","task":"KeycloakOrgEntityProvider:default:refresh"}

[Zaperex]

hmm that's very weird, if you try to run that image with the helm chart, it doesn't start the keycloak plugin? Can you try redeploying your helm chart with the same image you used locally?

[airedwin]

hmm that's very weird, if you try to run that image with the helm chart, it doesn't start the keycloak plugin?

ya, that's what it seems like... makes no sense lol

[airedwin]

docker push backstage:latest The push refers to repository [backstage] 5b6ad032d499: Layer already exists 980a005de253: Layer already exists 2716f38f144f: Layer already exists fe1070afb46d: Layer already exists 4917bed24219: Layer already exists fa9bd25e6cad: Layer already exists 8208215262de: Layer already exists 6281e481b4b4: Layer already exists 494cb968f6f5: Layer already exists b9c3c22c3fd6: Layer already exists 6f611743c2e1: Layer already exists 5d4427064ecc: Layer already exists

[airedwin]

i just repushed the image, it's the same

[Zaperex]

You changed the image field of the helm chart to refer to your image right? https://github.com/backstage/charts/blob/864c683f5f459ad021254e5dee093201bf7bfa8b/charts/backstage/values.yaml#L93-L102

[airedwin]

You changed the image field of the helm chart to refer to your image right? https://github.com/backstage/charts/blob/864c683f5f459ad021254e5dee093201bf7bfa8b/charts/backstage/values.yaml#L93-L102

yes

spec: containers:

• args:
  • '--config'
  • /app/app-config-from-configmap.yaml command:
  • node
  • packages/backend env:
  • name: APP_CONFIG_backend_listen_port value: '7007'
  • name: POSTGRES_HOST value: myserver
  • name: POSTGRES_PORT value: '5432'
  • name: POSTGRES_DATABASE value: backstage_dev
  • name: POSTGRES_USER value: backstage_admin
  • name: POSTGRES_PASSWORD valueFrom: secretKeyRef: key: password name: backstage-dev-database-secret
  • name: KEYCLOAK_CLIENTSECRET valueFrom: secretKeyRef: key: client-secret name: backstage-dev-keycloak-secret
  • name: APP_CONFIG_app_baseUrl value: 'https://backstage-dev'
  • name: APP_CONFIG_backend_baseUrl value: 'https://backstage-dev'
  • name: APP_CONFIG_backend_cors_origin value: 'https://backstage-dev'
  • name: NODE_TLS_REJECT_UNAUTHORIZED value: '0' image: 'backstage:latest' imagePullPolicy: Always

[airedwin]

i ran locally with

docker run -e POSTGRES_HOST=host.docker.internal -e POSTGRES_PORT=5432 -e POSTGRES_DATABASE=backstage -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD="" -e KEYCLOAK_CLIENTSECRET=mysecret -e APP_CONFIG_app_baseUrl=https://backstage-dev -e APP_CONFIG_backend_baseUrl=https://backstage-dev backstage:latest

[Zaperex]

I don't think backstage:latest would work? The correct format should be:

image:
  registry: <your registry>
  repository: <your repository>/backstage
  tag: latest

Can you perhaps inspect your backstage deployment pods to verify what the image being used is? I suspect it's just using the defaults.

[airedwin]

I don't think backstage:latest would work? The correct format should be:

image:
  registry: <your registry>
  repository: <your repository>/backstage
  tag: latest

Can you perhaps inspect your backstage deployment pods to verify what the image being used is? I suspect it's just using the defaults.

sorry, i redacted my private repo, but ya, i have the repo in there, it's like repo.com/library/backstage:latest

[airedwin]

ya, it's definitely on the server

[Zaperex]

Oh wait, can I ask where you're adding your keycloak backend configurations? I was able to "replicate" your logs by deleting my app-config.local.yaml with my keycloak backend configurations.

[Zaperex]

You would either need to inline your configurations in the appConfig field of the helm values when you install it. Ex:

appConfig:
  catalog:
    providers:
      keycloakOrg:
        default:
          baseUrl: http://localhost:8080
          realm: backstage
          loginRealm: backstage
          clientId: backstage
          clientSecret: <client-secret>
          schedule:
            frequency:
              minutes: 30
            timeout:
              minutes: 3
            initialDelay:
              seconds: 15

Alternatively, you can create a ConfigMap containing your custom-app-config.yaml configurations and add it to the extraAppConfig field. For example, in your values.yaml add:

extraAppConfig:
  - configMapRef: app-config-custom
    filename: custom-app-config.yaml

Then add the config map with your configurations, change the values to fit your needs

kind: ConfigMap
apiVersion: v1
metadata:
  name: app-config-custom

data:
  custom-app-config.yaml: |
    catalog:
      providers:
        keycloakOrg:
          default:
            baseUrl: http://localhost:8080
            realm: backstage
            loginRealm: backstage
            clientId: backstage
            clientSecret: <client-secret>
            schedule:
              frequency:
                minutes: 30
              timeout:
                minutes: 3
              initialDelay:
                seconds: 15

[airedwin]

i'm pretty sure i did all that

[airedwin]

$ pwd
/app
$ ls
app-config-from-configmap.yaml  app-config.yaml  node_modules  package.json  packages  yarn.lock
$ cat app-config-from-configmap.yaml
app:
  baseUrl: https://backstage-dev
  title: Backstage
auth:
  environment: production
  providers:
    oauth2Proxy:
      signIn:
        resolvers:
        - resolver: forwardedUserMatchingUserEntityName
backend:
  baseUrl: https://backstage-dev
  database:
    client: pg
    connection:
      database: ${POSTGRES_DATABASE}
      host: ${POSTGRES_HOST}
      password: ${POSTGRES_PASSWORD}
      port: ${POSTGRES_PORT}
      user: ${POSTGRES_USER}
    pluginDivisionMode: schema
  listen:
    port: :7007
catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  keycloakOrg:
    default:
      baseUrl: https://keycloak-dev/auth
      clientId: backstage-confidential
      clientSecret: ${KEYCLOAK_CLIENTSECRET}
      loginRealm: realm
      realm: realm
      schedule:
        frequency:
          minutes: 30
        initialDelay:
          seconds: 15
        timeout:
          minutes: 3
  locations:
  - target: ../../examples/entities.yaml
    type: file
  - rules:
    - allow:
      - Template
    target: ../../examples/template/template.yaml
    type: file
  - rules:
    - allow:
      - User
      - Group
    target: ../../examples/org.yaml
    type: file
  providers: null
  rules:
  - allow:
    - Component
    - System
    - API
    - Resource
    - Location
enabled:
  keycloak: true
techdocs:
  builder: local
  generator:
    runIn: docker
  publisher:
    type: local

unless this isn't the file that is being used

[Zaperex]

ah you misconfigured it in your app-config-from-configmap.yaml. you used

catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  keycloakOrg:
    default:
      baseUrl: https://keycloak-dev/auth
      clientId: backstage-confidential
      clientSecret: ${KEYCLOAK_CLIENTSECRET}
      loginRealm: realm
      realm: realm
      schedule:
        frequency:
          minutes: 30
        initialDelay:
          seconds: 15
        timeout:
          minutes: 3

instead of doing catalog.providers.keycloakOrg

[airedwin]

i just noticed that too, i think it's in my helm somewhere that is missing an indentation lol, thanks!

[Zaperex]

let me know if it works, if it does I'll close the issue.

[airedwin]

it's working, i think we should delete this, user error lol

[airedwin]

@Zaperex I got this working but it doesn't seem like the provider can match up an identity with a user in the catalog. I assume it's a resolver issue? I'm using:

auth:
  providers:
    oauth2Proxy:
      signIn:
        resolvers:
          - resolver: forwardedUserMatchingUserEntityName

should I be using something different? the users are imported correctly

{"class":"KeycloakOrgEntityProvider","level":"info","message":"Reading Keycloak users and groups","plugin":"catalog","service":"backstage","taskId":"KeycloakOrgEntityProvider:default:refresh","taskInstanceId":"bed5788f-c893-447c-a659-43976c9bebd6"}
{"class":"KeycloakOrgEntityProvider","level":"info","message":"Read 9 Keycloak users and 4 Keycloak groups in 0.2 seconds. Committing...","plugin":"catalog","service":"backstage","taskId":"KeycloakOrgEntityProvider:default:refresh","taskInstanceId":"bed5788f-c893-447c-a659-43976c9bebd6"}
{"class":"KeycloakOrgEntityProvider","level":"info","message":"Committed 9 Keycloak users and 4 Keycloak groups in 0.0 seconds.","plugin":"catalog","service":"backstage","taskId":"KeycloakOrgEntityProvider:default:refresh","taskInstanceId":"bed5788f-c893-447c-a659-43976c9bebd6"}

this is the error i'm getting in the log

{"level":"info","message":"::1 - - [10/Sep/2024:21:01:51 +0000] \"GET /api/catalog/entities/by-name/User/default/1f2b03f8-bbe0-4ae2-8640-52e56873962e HTTP/1.1\" 404 281 \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"rootHttpRouter","type":"incomingRequest"}
{"level":"error","message":"Request failed with status 500 Failed to sign-in, unable to resolve user identity","service":"rootHttpRouter","stack":"Error: Failed to sign-in, unable to resolve user identity\n    at /app/node_modules/@backstage/plugin-auth-node/dist/index.cjs.js:860:11\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async Object.refresh (/app/node_modules/@backstage/plugin-auth-node/dist/index.cjs.js:952:24)","type":"errorHandler"}

it looks like it's querying on the keycloak id which is in metadata.annotations.keycloak.org/id, do i need to write a custom resolver for this?

[airedwin]

this was the data put into the catalog by your plugin

[Zaperex]

Hmm that's weird, it seems the issue on your end is that the x-forwarded-user header seems be resolving to be the id of the keycloak user instead of the username of the user since the name of the user entity that gets queried is determined by this header value: https://github.com/backstage/backstage/blob/f094dfd54a293bfe03cd00cb6dbd7f562cc26a8b/plugins/auth-backend-module-oauth2-proxy-provider/src/resolvers.ts#L31.

This may require configuring your keycloak instance or proxy to return the correct x-forwarded-user header, or a custom resolver might be needed to match to the id annotation instead?

[airedwin]

Ya,.I think so too, thanks. I'm using the default oauth2proxy sidecar for backstage as part of their helm chart, I'll see if I can configure that forwarded header

[airedwin]

Actually, I think that is correct, the user header is id and there is a preferred-username header that is the username... would it make sense to change your plugin to put the id in the catalog name field?

[airedwin]

Or maybe it's a keycloak client config of what to put in the user claim

[Zaperex]

There is also the option of installing a transformer module to set the name of the entity to the id instead of the username.

[airedwin]

There is also the option of installing a transformer module to set the name of the entity to the id instead of the username.

Thanks, is there any documentation on how to build the entity to return?

[airedwin]

Nevermind, I foudn it https://github.com/janus-idp/backstage-plugins/blob/main/plugins/keycloak-backend/src/lib/read.ts

[airedwin]

I'm not sure if i'm doing this right, but i created a custom module for the usertransformer and i'm getting an error

/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648
              provides: Array.from(moduleInit.consumes).map((c) => c.id)
                                                                     ^
TypeError: Cannot read properties of undefined (reading 'id')
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648:70
    at Array.map (<anonymous>)
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1648:57
    at Array.map (<anonymous>)
    at /app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1642:33
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 5)
    at async #doStart (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1633:5)
    at async BackendInitializer.start (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1562:5)
    at async BackstageBackend.start (/app/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1762:5)

I'm assuming the module isn't imported right this is my module

import { createBackendModule } from '@backstage/backend-plugin-api';
import type UserRepresentation from '@keycloak/keycloak-admin-client/lib/defs/userRepresentation';
import { UserEntity } from '@backstage/catalog-model';

import {
  keycloakTransformerExtensionPoint,
  UserTransformer,
  GroupRepresentationWithParentAndEntity,
  KeycloakTransformerExtensionPoint,
} from '@janus-idp/backstage-plugin-keycloak-backend';

export const catalogModuleCustomUserTransformer = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'custom-user-transformer',
  register(reg) {
    reg.registerInit({
      deps: { keycloak: keycloakTransformerExtensionPoint },
      async init({
        keycloak,
      }: {
        keycloak: KeycloakTransformerExtensionPoint;
      }) {
        keycloak.setUserTransformer(customUserTransformer);
      },
    });
  },
});

const customUserTransformer: UserTransformer = async (
  entity: UserEntity,
  user: UserRepresentation,
  realm: string,
  groups: GroupRepresentationWithParentAndEntity[],
) => {
  return {
    ...entity,
    metadata: {
      name: user.id!,
    },
  };
};

and i imported it like this

backend.add(import('@janus-idp/backstage-plugin-keycloak-backend/alpha'));
backend.add(import('@internal/backstage-plugin-catalog-backend-module-custom-user-transformer'));

[airedwin]

Updating fixed it