@roadiehq/catalog-backend-module-okta/new-backend not working properly

[z-sourcecode]

new backend plugin does not pull users and groups from OKTA.

Expected Behavior

Pulling Users and groups to backstage schema

Current Behavior

plugin seems to run but return 0 users and 0 groups

Steps to Reproduce

app-config.yaml:

catalog:
  providers:
    okta:
      - orgUrl: ${OKTA_AUDIENCE}
        token: ${OKTA_API_TOKEN}
        frequency: { minutes: 1 }
        timeout: { minutes: 3 }

/backstage/packages/backend/src/index.ts

import { EntityProviderFactory, oktaCatalogBackendEntityProviderFactoryExtensionPoint } from '@roadiehq/catalog-backend-module-okta/new-backend';

// Loading Okta catalog
export const oktaCatalogBackendModule = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'okta-entity-provider-custom',
  register(env) {
    env.registerInit({
      deps: {
        provider: oktaCatalogBackendEntityProviderFactoryExtensionPoint,
        logger: coreServices.logger,
      },
      async init({ provider, logger }) {
        const factory: EntityProviderFactory = (oktaConfig: Config) =>
          OktaOrgEntityProvider.fromConfig(oktaConfig, {
            logger: loggerToWinstonLogger(logger),
            userNamingStrategy: 'strip-domain-email',
            groupNamingStrategy: 'kebab-case-name',
          });

        provider.setEntityProviderFactory(factory);
      },
    });
  },
});

backend.add(oktaCatalogBackendModule);
backend.add(import('@roadiehq/catalog-backend-module-okta/new-backend'));

Logs: [1] 2024-08-07T13:48:20.752Z auth info Configuring auth provider: okta [1] 2024-08-07T13:48:20.757Z catalog info Task worker starting: okta-entity-provider-okta-org:all, {"version":2,"cadence":"PT1M","timeoutAfterDuration":"PT3M"} task=okta-entity-provider-okta-org:all [1] 2024-08-07T13:48:20.759Z catalog info Performing database migration [1] 2024-08-07T13:48:20.765Z catalog error Error: Not initialized task=okta-entity-provider-okta-org:all [1] 2024-08-07T13:48:20.812Z search info Added DefaultCatalogCollatorFactory collator factory for type software-catalog [1] 2024-08-07T13:49:20.833Z catalog info Providing user and group resources from okta [1] 2024-08-07T13:49:20.834Z catalog info Found 0, pruning the empty ones [1] 2024-08-07T13:49:20.841Z catalog info Finished providing 0 user and 0 group resources from okta

Possible Solution

Context

While initiating backstage

Your Environment

"dependencies": { "@aws/backstage-plugin-catalog-backend-module-aws-apps-entities-processor": "0.3.4", "@aws/plugin-aws-apps-backend-for-backstage": "0.3.4", "@aws/plugin-scaffolder-backend-aws-apps-for-backstage": "0.3.4", "@backstage/backend-common": "^0.23.3", "@backstage/backend-defaults": "^0.4.0", "@backstage/backend-tasks": "^0.5.27", "@backstage/config": "^1.2.0", "@backstage/plugin-app-backend": "^0.3.71", "@backstage/plugin-auth-backend": "^0.22.9", "@backstage/plugin-auth-backend-module-github-provider": "^0.1.19", "@backstage/plugin-auth-backend-module-guest-provider": "^0.1.8", "@backstage/plugin-auth-node": "^0.4.17", "@backstage/plugin-catalog-backend": "^1.24.0", "@backstage/plugin-catalog-backend-module-github": "^0.6.5", "@backstage/plugin-catalog-backend-module-gitlab": "^0.3.21", "@backstage/plugin-catalog-backend-module-logs": "^0.0.1", "@backstage/plugin-catalog-backend-module-scaffolder-entity-model": "^0.1.20", "@backstage/plugin-permission-backend": "^0.5.46", "@backstage/plugin-permission-backend-module-allow-all-policy": "^0.1.19", "@backstage/plugin-permission-common": "^0.8.0", "@backstage/plugin-permission-node": "^0.8.0", "@backstage/plugin-proxy-backend": "^0.5.3", "@backstage/plugin-scaffolder-backend": "^1.23.0", "@backstage/plugin-search-backend": "^1.5.14", "@backstage/plugin-search-backend-module-catalog": "^0.1.28", "@backstage/plugin-search-backend-module-pg": "^0.5.32", "@backstage/plugin-search-backend-module-techdocs": "^0.1.27", "@backstage/plugin-search-backend-node": "^1.2.27", "@backstage/plugin-techdocs-backend": "^1.10.9", "@immobiliarelabs/backstage-plugin-gitlab-backend": "^6.6.0", "@roadiehq/catalog-backend-module-okta": "^0.10.0", "@roadiehq/scaffolder-backend-module-utils": "^1.17.1", "app": "link:../app", "better-sqlite3": "^9.0.0", "node-gyp": "^10.0.0", "pg": "^8.11.3", "winston": "^3.2.1" }, "devDependencies": { "@backstage/cli": "^0.26.11", "@types/express": "^4.17.6", "@types/express-serve-static-core": "^4.17.5", "@types/luxon": "^2.0.4" }, "files": [ "dist" ] }

[GersonTrj]

I am experiencing the same issue!

@Xantier do you know if this plugin is going to be maintained and/or do you have an estimate on when you could check this? I am asking because I am considering if we should use this plugin.

Do you have any working example or documentation using the new backend system?

[z-sourcecode]

Until a solution is provided this code worked for me:

/backstage/packages/backend/src/index.ts

// Loading OKTA users and groups - Legacy code:
backend.add(legacyPlugin('roadiehq-okta-catalog', import('./plugins/catalog')));

catalog.ts

import { CatalogBuilder } from '@backstage/plugin-catalog-backend';
import { Router } from 'express';
import { PluginEnvironment } from '../types';
import { OktaOrgEntityProvider } from '@roadiehq/catalog-backend-module-okta';

export default async function createPlugin(
  env: PluginEnvironment,
): Promise<Router> {
  const builder = await CatalogBuilder.create(env);
  const orgProvider = OktaOrgEntityProvider.fromConfig(env.config, {
    logger: env.logger,
    userNamingStrategy: 'strip-domain-email',
    groupNamingStrategy: 'kebab-case-name',
  });

  builder.addEntityProvider(orgProvider);

  const { processingEngine, router } = await builder.build();
  orgProvider.run();
  await processingEngine.start();
  return router;
}

[awsjim]

The new-backend passes the oktaConfig (i.e. "catalog.providers.okta") to the EntityProviderFactory: https://github.com/RoadieHQ/roadie-backstage-plugins/blob/04a21af5938c208b82e6e5857500d9b6d308eb18/plugins/backend/catalog-backend-module-okta/src/module.ts#L73

However, OktaOrgEntityProvider.fromConfig(...) expects the full Config and tries again to get the "catalog.providers.okta" sub-config: https://github.com/RoadieHQ/roadie-backstage-plugins/blob/04a21af5938c208b82e6e5857500d9b6d308eb18/plugins/backend/catalog-backend-module-okta/src/providers/OktaOrgEntityProvider.ts#L78-L80

A workaround can be to pass the full config to the fromConfig() method

# backstage/packages/backend/src/index.ts

import { coreServices } from '@backstage/backend-plugin-api';
import { OktaOrgEntityProvider, EntityProviderFactory, oktaCatalogBackendEntityProviderFactoryExtensionPoint } from '@roadiehq/catalog-backend-module-okta/new-backend';

...

backend.add(import('@roadiehq/catalog-backend-module-okta/new-backend'));
export const oktaCatalogBackendModule = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'okta-entity-provider-custom',
  register(env) {
    env.registerInit({
      deps: {
        provider: oktaCatalogBackendEntityProviderFactoryExtensionPoint,
        logger: coreServices.logger,
        config: coreServices.rootConfig,
      },
      async init({ provider, logger, config }) {
        const factory: EntityProviderFactory = (oktaConfig: Config) => 
          OktaOrgEntityProvider.fromConfig(config, {
            logger: loggerToWinstonLogger(logger),
            userNamingStrategy: 'strip-domain-email',
            groupNamingStrategy: 'kebab-case-name',
          });

        provider.setEntityProviderFactory(factory);
      },
    });
  },
});

backend.add(oktaCatalogBackendModule);

[GersonTrj]

@awsjim THANK YOU so much, it works indeed after passing the rootConfig!!

[hexionas]

In my case the workaround from @awsjim works, but strangely it first fails with:

2024-08-09T14:55:26.037Z catalog info Task worker starting: okta-entity-provider-okta-org:all, {"version":2,"cadence":"PT1M","timeoutAfterDuration":"PT3M"} task=okta-entity-provider-okta-org:all
2024-08-09T14:55:26.040Z catalog error Error: Not initialized task=okta-entity-provider-okta-org:all
2024-08-09T14:55:26.040Z catalog debug task: okta-entity-provider-okta-org:all will next occur around 2024-08-09T16:56:26.040+02:00 task=okta-entity-provider-okta-org:all

And on the next scheduled iteration the groups and users get ingested.

2024-08-09T14:56:26.065Z catalog info Providing user and group resources from okta
2024-08-09T14:56:26.451Z catalog debug task: catalog_orphan_cleanup will next occur around 2024-08-09T16:56:56.451+02:00 task=catalog_orphan_cleanup
2024-08-09T14:56:27.872Z catalog info Found 9 groups in okta, pruning the empty ones
2024-08-09T14:56:27.893Z catalog info Finished providing 81 user and 9 group resources from okta
2024-08-09T14:56:27.893Z catalog debug task: okta-entity-provider-okta-org:all will next occur around 2024-08-09T16:57:27.893+02:00 task=okta-entity-provider-okta-org:all

[awsjim]

I noticed the "Error: not initialized" task error also. I think it's only occurring on the first run of the task because the connection is not yet created at first run. https://github.com/RoadieHQ/roadie-backstage-plugins/blob/04a21af5938c208b82e6e5857500d9b6d308eb18/plugins/backend/catalog-backend-module-okta/src/providers/OktaOrgEntityProvider.ts#L129-L132

I don't think it causes any harm, but it does look like it means you have to wait for the "frequency" defined in your configuration before the users/groups will be populated the first time. This is just my reading of the code and defer to the maintainers for the real answers and recommendations : ).

[chauvm]

Can someone help providing the full new backend instructions for this plugin? I've followed @awsjim 's instructions to use the rootConfig and get passed the Not initialized error, but my task seems to hit a different error.

Debug log:

catalog info Providing ***er and group resources from okta
catalog error OktaApiError: Okta HTTP 400 undefined  task=okta-entity-provider-okta-org:all
catalog debug task: okta-entity-provider-okta-org:all will next occur around 2024-08-12T22:16:34.138+00:00 task=okta-entity-provider-okta-org:all

This is what I have done, please help me check if I miss any steps for the new backend setup:

// install the module in backend
yarn --cwd packages/backend add @roadiehq/catalog-backend-module-okta

// apps/backstage/packages/backend/src/index.ts
import { coreServices, createBackendModule } from '@backstage/backend-plugin-api';
import { OktaOrgEntityProvider } from '@roadiehq/catalog-backend-module-okta';
import { EntityProviderFactory, oktaCatalogBackendEntityProviderFactoryExtensionPoint } from '@roadiehq/catalog-backend-module-okta/new-backend';
import { loggerToWinstonLogger } from '@backstage/backend-common';
import { Config } from '@backstage/config';

backend.add(import('@roadiehq/catalog-backend-module-okta/new-backend'));
export const oktaCatalogBackendModule = createBackendModule({
  pluginId: 'catalog',
  moduleId: 'okta-entity-provider-custom',
  register(env) {
    env.registerInit({
      deps: {
        provider: oktaCatalogBackendEntityProviderFactoryExtensionPoint,
        logger: coreServices.logger,
        config: coreServices.rootConfig,
      },
      async init({ provider, logger, config }) {
        const factory: EntityProviderFactory = (_oktaConfig: Config) =>
          OktaOrgEntityProvider.fromConfig(config, {
            logger: loggerToWinstonLogger(logger),
            userNamingStrategy: 'strip-domain-email',
            groupNamingStrategy: 'kebab-case-name',
          });

        provider.setEntityProviderFactory(factory);
      },
    });
  },
});

backend.add(oktaCatalogBackendModule);

// app-config.yaml
catalog:
  providers:
    okta:
      - orgUrl: 'https://zipline.okta.com'
        oauth:
          clientId: ${OKTA_OAUTH_CLIENT_ID}
          keyId: ${OKTA_OAUTH_KEY_ID}  # set this if provided PEM key as privateKey
          privateKey: ${OKTA_OAUTH_PRIVATE_KEY}  # PEM key or JWT in JSON
        frequency: { minutes: 5 }
        timeout: { minutes: 2 }

Update: I checked my Okta Oauth app's logs and it shows OIDC token request: FAILURE: unauthorized_client every time the task runs. I followed instructions in the README to create the Oauth app with okta.groups.read and okta.users.read scopes. I have tried setting oauth.privateKey to JWT and PEM (and provided keyId), but got the same error in both cases (OktaApiError: Okta HTTP 400 undefined in Backstage and unauthorized_client in Okta Oauth app).

[Xantier]

We started the process of upgrading our plugins to support the new system so this should be addressed soon.

[GersonTrj]

@Xantier do you have an estimate on this since it's been 3 weeks already?

[Xantier]

We haven't gotten to this one yet but it is still on the top end of the backlog at the moment