🐛 Bug Report: MicrosoftGraphOrgEntityProvider doesn't work

MarkPorohnja commented 4 months ago

📜 Description

After successfully testing microsoft auth provider, we attempted to utilize MicrosoftGraphOrgEntityProvider to populate Backstage catalog, following the documentation. App registration has all necessary permissions. But expected functionality is not being achieved. Plugin docs: https://github.com/backstage/backstage/blob/master/plugins/catalog-backend-module-msgraph/README.md

👍 Expected behavior

Provider is able to find users and groups to populate Backstage catalog.

👎 Actual Behavior with Screenshots

From documentation and logs, It's unclear where we made a mistake. The absence of the log message Reading msgraph users and groups suggests that the provider was not registered.

logs:

Loading config from MergedConfigSource{FileConfigSource{path="/app/app-config-from-configmap.yaml"}, EnvConfigSource{count=1}}
{"level":"info","message":"Found 3 new secrets in config that will be redacted","service":"backstage"}
{"level":"info","message":"Listening on :7007","service":"rootHttpRouter"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"proxy","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"proxy","service":"backstage"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"permission","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"permission","service":"backstage"}
{"level":"info","message":"Configuring \"database\" as KeyStore provider","plugin":"auth","service":"backstage"}
{"level":"info","message":"Creating Local publisher for TechDocs","plugin":"techdocs","service":"backstage"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"techdocs","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"techdocs","service":"backstage"}
{"level":"info","message":"Starting scaffolder with the following actions enabled fetch:plain, fetch:plain:file, fetch:template, debug:log, debug:wait, catalog:register, catalog:fetch, catalog:write, fs:delete, fs:rename","plugin":"scaffolder","service":"backstage"}
{"level":"info","message":"Added DefaultCatalogCollatorFactory collator factory for type software-catalog","plugin":"search","service":"backstage"}
{"level":"info","message":"Added DefaultTechDocsCollatorFactory collator factory for type techdocs","plugin":"search","service":"backstage"}
{"level":"info","message":"Starting all scheduled search tasks.","plugin":"search","service":"backstage"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"search","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"search","service":"backstage"}
{"level":"info","message":"Serving static app content from /app/packages/app/dist","plugin":"app","service":"backstage"}
{"level":"info","message":"Configuring auth provider: microsoft","plugin":"auth","service":"backstage"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"auth","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"auth","service":"backstage"}
{"level":"info","message":"Performing database migration","plugin":"catalog","service":"backstage"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"scaffolder","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"scaffolder","service":"backstage"}
{"level":"info","message":"Injecting env config into module-backstage.1e48c328.js","plugin":"app","service":"backstage"}
{"level":"info","message":"Task worker starting: search_index_software_catalog, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_software_catalog"}
{"level":"info","message":"Task worker starting: search_index_techdocs, {\"version\":2,\"cadence\":\"PT10M\",\"initialDelayDuration\":\"PT3S\",\"timeoutAfterDuration\":\"PT15M\"}","plugin":"search","service":"backstage","task":"search_index_techdocs"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"catalog","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"catalog","service":"backstage"}
{"level":"info","message":"Storing 287 updated assets and 0 new assets","plugin":"app","service":"backstage"}
{"level":"debug","message":"Running 1 plugin startup tasks...","plugin":"app","service":"backstage"}
{"level":"debug","message":"Plugin startup hook succeeded","plugin":"app","service":"backstage"}
{"level":"debug","message":"Running 1 startup tasks...","service":"backstage"}
{"level":"debug","message":"Startup hook succeeded","service":"backstage"}
{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog via DefaultCatalogCollatorFactory","plugin":"search","service":"backstage"}
{"documentType":"techdocs","level":"info","message":"Collating documents for techdocs via DefaultTechDocsCollatorFactory","plugin":"search","service":"backstage"}
{"level":"info","message":"::1 - - [22/Apr/2024:12:43:30 +0000] \"GET /api/catalog/entities?limit=500&offset=0 HTTP/1.1\" 200 2 \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"rootHttpRouter","type":"incomingRequest"}
{"documentType":"software-catalog","level":"warn","message":"Index for software-catalog was not created: indexer received 0 documents","plugin":"search","service":"backstage"}
{"documentType":"software-catalog","level":"info","message":"Collating documents for software-catalog succeeded","plugin":"search","service":"backstage"}
{"level":"info","message":"::1 - - [22/Apr/2024:12:43:30 +0000] \"GET /api/catalog/entities?limit=500&filter=metadata.annotations.backstage.io%2Ftechdocs-ref&offset=0 HTTP/1.1\" 200 2 \"-\" \"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)\"","service":"rootHttpRouter","type":"incomingRequest"}
{"documentType":"techdocs","level":"warn","message":"Index for techdocs was not created: indexer received 0 documents","plugin":"search","service":"backstage"}
{"documentType":"techdocs","level":"info","message":"Collating documents for techdocs succeeded","plugin":"search","service":"backstage"}

👟 Reproduction steps

catalog.ts

import { CatalogBuilder } from '@backstage/plugin-catalog-backend';
import { ScaffolderEntitiesProcessor } from '@backstage/plugin-catalog-backend-module-scaffolder-entity-model';
import { Router } from 'express';
import { PluginEnvironment } from '../src/types';
import { MicrosoftGraphOrgEntityProvider } from '@backstage/plugin-catalog-backend-module-msgraph';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const builder = CatalogBuilder.create(env);
  builder.addProcessor(new ScaffolderEntitiesProcessor());

  builder.addEntityProvider(
    MicrosoftGraphOrgEntityProvider.fromConfig(env.config, {
      logger: env.logger,
      schedule: env.scheduler.createScheduledTaskRunner({
          frequency: { hours: 1 },
          timeout: { minutes: 50 },
          initialDelay: { seconds: 15},
      })
    }),
  );

  const { processingEngine, router } = await builder.build();

  await processingEngine.start();
  return router;
}

app-config.yaml

    catalog:
      providers:
        microsoftGraphOrg:
          providerId:
            target: https://graph.microsoft.com/v1.0
            authority: https://login.microsoftonline.com
            tenantId: **REDACTED**
            clientId: **REDACTED**
            clientSecret: **REDACTED**
            user:
              filter: accountEnabled eq true and userType eq 'member'
            group:
              filter: displayName eq '**REDACTED**'

app registration permissions:

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

OS: Linux 5.15.146.1-microsoft-standard-WSL2 - linux/x64 node: v20.11.1 yarn: 1.22.22 cli: 0.26.2 (installed) backstage: 1.25.0

Dependencies: @backstage/app-defaults 1.5.3 @backstage/backend-app-api 0.6.2, 0.7.0 @backstage/backend-common 0.21.6, 0.21.7 @backstage/backend-defaults 0.2.16 @backstage/backend-dev-utils 0.1.4 @backstage/backend-openapi-utils 0.1.9 @backstage/backend-plugin-api 0.6.16, 0.6.17 @backstage/backend-tasks 0.5.21, 0.5.22 @backstage/catalog-client 1.6.3, 1.6.4 @backstage/catalog-model 1.4.5 @backstage/cli-common 0.1.13 @backstage/cli-node 0.2.4, 0.2.5 @backstage/cli 0.26.2 @backstage/config-loader 1.7.0, 1.8.0 @backstage/config 1.2.0 @backstage/core-app-api 1.12.3 @backstage/core-compat-api 0.2.3 @backstage/core-components 0.14.3 @backstage/core-plugin-api 1.9.1 @backstage/e2e-test-utils 0.1.1 @backstage/errors 1.2.4 @backstage/eslint-plugin 0.1.6 @backstage/frontend-plugin-api 0.6.3 @backstage/integration-aws-node 0.1.12 @backstage/integration-react 1.1.25 @backstage/integration 1.10.0, 1.9.1 @backstage/plugin-api-docs 0.11.3 @backstage/plugin-app-backend 0.3.64 @backstage/plugin-app-node 0.1.16 @backstage/plugin-auth-backend-module-atlassian-provider 0.1.8 @backstage/plugin-auth-backend-module-aws-alb-provider 0.1.8 @backstage/plugin-auth-backend-module-gcp-iap-provider 0.2.11 @backstage/plugin-auth-backend-module-github-provider 0.1.13 @backstage/plugin-auth-backend-module-gitlab-provider 0.1.13 @backstage/plugin-auth-backend-module-google-provider 0.1.13 @backstage/plugin-auth-backend-module-microsoft-provider 0.1.11 @backstage/plugin-auth-backend-module-oauth2-provider 0.1.13 @backstage/plugin-auth-backend-module-oauth2-proxy-provider 0.1.9 @backstage/plugin-auth-backend-module-oidc-provider 0.1.7 @backstage/plugin-auth-backend-module-okta-provider 0.0.9 @backstage/plugin-auth-backend 0.22.3 @backstage/plugin-auth-node 0.4.11, 0.4.12 @backstage/plugin-auth-react 0.0.3 @backstage/plugin-catalog-backend-module-msgraph 0.5.25 @backstage/plugin-catalog-backend-module-scaffolder-entity-model 0.1.14 @backstage/plugin-catalog-backend-module-unprocessed 0.4.4 @backstage/plugin-catalog-backend 1.21.0 @backstage/plugin-catalog-common 1.0.22 @backstage/plugin-catalog-graph 0.4.3 @backstage/plugin-catalog-import 0.10.9 @backstage/plugin-catalog-node 1.11.0, 1.11.1 @backstage/plugin-catalog-react 1.11.2 @backstage/plugin-catalog-unprocessed-entities-common 0.0.1 @backstage/plugin-catalog 1.18.2 @backstage/plugin-events-node 0.3.2 @backstage/plugin-github-actions 0.6.14 @backstage/plugin-org 0.6.23 @backstage/plugin-permission-backend-module-allow-all-policy 0.1.13 @backstage/plugin-permission-backend 0.5.40 @backstage/plugin-permission-common 0.7.13 @backstage/plugin-permission-node 0.7.27, 0.7.28 @backstage/plugin-permission-react 0.4.21 @backstage/plugin-proxy-backend 0.4.14 @backstage/plugin-scaffolder-backend-module-azure 0.1.8 @backstage/plugin-scaffolder-backend-module-bitbucket-cloud 0.1.6 @backstage/plugin-scaffolder-backend-module-bitbucket-server 0.1.6 @backstage/plugin-scaffolder-backend-module-bitbucket 0.2.6 @backstage/plugin-scaffolder-backend-module-gerrit 0.1.8 @backstage/plugin-scaffolder-backend-module-gitea 0.1.6 @backstage/plugin-scaffolder-backend-module-github 0.2.6 @backstage/plugin-scaffolder-backend-module-gitlab 0.3.2 @backstage/plugin-scaffolder-backend 1.22.3 @backstage/plugin-scaffolder-common 1.5.1 @backstage/plugin-scaffolder-node 0.4.2 @backstage/plugin-scaffolder-react 1.8.3 @backstage/plugin-scaffolder 1.19.2 @backstage/plugin-search-backend-module-catalog 0.1.21 @backstage/plugin-search-backend-module-techdocs 0.1.21 @backstage/plugin-search-backend-node 1.2.20 @backstage/plugin-search-backend 1.5.6 @backstage/plugin-search-common 1.2.11 @backstage/plugin-search-react 1.7.9 @backstage/plugin-search 1.4.9 @backstage/plugin-tech-radar 0.7.2 @backstage/plugin-techdocs-backend 1.10.3 @backstage/plugin-techdocs-module-addons-contrib 1.1.8 @backstage/plugin-techdocs-node 1.12.2 @backstage/plugin-techdocs-react 1.2.2 @backstage/plugin-techdocs 1.10.3 @backstage/plugin-user-settings 0.8.4 @backstage/release-manifests 0.0.11 @backstage/test-utils 1.5.3 @backstage/theme 0.5.2 @backstage/types 1.1.1 @backstage/version-bridge 1.0.7

👀 Have you spent some time to check if this bug has been raised before?

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?

None

MarkPorohnja commented 3 months ago

Seems that the task is not scheduled. Logs contain information about scheduling 2 default task and nothing related to entity provider.

Parsifal-M commented 3 months ago

Hey! 👋

Looks like, based on your version, you should be on the new backend system? If that is the case you shouldn't have a catalog.ts file.

In your packages/backend/src/index.ts you'd import it like this

backend.add(import('@backstage/plugin-catalog-backend-module-msgraph/alpha'));

I think your app-config looks okay.

You may need to read these docs if you are not sure about the new backend part: https://backstage.io/docs/backend-system/building-backends/migrating

Hope this helps?

MarkPorohnja commented 3 months ago

Yes, thank you. Could you please update the documentation of Microsoft Entra Tenant Data page https://backstage.io/docs/integrations/azure/org/ It is still based on old backend and there is no reference on "migrating" page

MarkPorohnja commented 3 months ago

how to make transformations with new backend if there is no catalog.ts file?

Parsifal-M commented 3 months ago

how to make transformations with new backend if there is no catalog.ts file?

Hey 👋

So what we do is in our packages/backend/src folder we have a plugins folder (which we should probably rename at some point) that has a transformers.ts file in it where we keep and export the user transformer and the group transformer we use.

Then in the index.ts file (from above) we do this:

backend.add(
  createBackendModule({
    pluginId: 'catalog',
    moduleId: 'microsoft-graph-extensions',
    register(env) {
      env.registerInit({
        deps: {
          microsoftGraphTransformers: microsoftGraphOrgEntityProviderTransformExtensionPoint,
        },
        async init({ microsoftGraphTransformers }) {
          microsoftGraphTransformers.setUserTransformer(myUserTransformer);
          microsoftGraphTransformers.setGroupTransformer(myGroupTransformer);
        },
      });
    },
  }),
);

so in the end you'd have the first import which I mentioned initially, then this second one as well for the transformers, this is using what is referred to in the docs as an extension point 👍 hope this helps?

MarkPorohnja commented 3 months ago

Thank you 👍

Parsifal-M commented 3 months ago

Yes, thank you. Could you please update the documentation of Microsoft Entra Tenant Data page https://backstage.io/docs/integrations/azure/org/ It is still based on old backend and there is no reference on "migrating" page

Just FYI also, there is this initiative #24195 to update all the docs for the new backend :)

backstage / backstage