Support IDM-only deployments

debasisdwivedy commented 1 month ago

Hi,

Does FRODO utility support IDM import/export utility like it does for AM.

I can see in the below for frodo frodo idm --help

`Manage IDM configuration.

Options: -h, --help Help

Commands: count Count number of managed objects of a given type. export Export IDM configuration objects. help [command] display help for command import Import IDM configuration objects. list List all IDM configuration objects.`

But there is no explanation how todo that.

Could someone provide me an example command for an import/export of IDM config using FRODO if its possible.

Regards, Debasis

vscheuber commented 1 month ago

@debasisdwivedy thank you for your request! You can invoke detail help on each command and sub-command using the -h option or the help command:

frodo idm export -h or frodo idm help export, which both return the following help:

Usage: frodo idm export [options] [host] [realm] [username] [password]

Export IDM configuration objects.

Arguments:
  host                                 Access Management base URL, e.g.: https://cdk.iam.example.com/am. To use a connection profile, just specify a unique substring.
  realm                                Realm. Specify realm as '/' for the root realm or 'realm' or '/parent/child' otherwise. (default: "alpha" for Identity Cloud tenants, "/" otherwise.)
  username                             Username to login with. Must be an admin user with appropriate rights to manage authentication journeys/trees.
  password                             Password.

Options:
  -a, --all                            Export all IDM configuration objects into a single file in directory -D. Ignored with -N.
  -A, --all-separate                   Export all IDM configuration objects into separate JSON files in directory -D. Ignored with -N, and -a.
  --curlirize                          Output all network calls in curl format.
  -D, --directory <directory>          Set the working directory.
  --debug                              Debug output during command execution. If specified, may or may not produce additional output helpful for troubleshooting.
  -e, --env-file [envfile]             Name of the env file. Ignored with -A.
  -E, --entities-file [entities-file]  Name of the entity file. Ignored with -A.
  -f, --file [file]                    Export file (or directory name if exporting mappings separately). Ignored with -A.
  --flush-cache                        Flush token cache.
  -h, --help                           Help
  -k, --insecure                       Allow insecure connections when using SSL/TLS. Has no effect when using a network proxy for https (HTTPS_PROXY=http://<host>:<port>), in that case the proxy must provide this capability. (default:
                                       Don't allow insecure connections)
  -m, --type <type>                    Override auto-detected deployment type. Valid values for type:
                                       classic:  A classic Access Management-only deployment with custom layout and configuration.
                                       cloud:    A ForgeRock Identity Cloud environment.
                                       forgeops: A ForgeOps CDK or CDM deployment.
                                       The detected or provided deployment type controls certain behavior like obtaining an Identity Management admin token or not and whether to export/import referenced email templates or how to walk
                                       through the tenant admin login flow of Identity Cloud and handle MFA (choices: "classic", "cloud", "forgeops")
  -N, --name <name>                    Config entity name. E.g. "managed", "sync", "provisioner-<connector-name>", etc.
  --no-cache                           Disable token cache for this operation.
  -s, --separate-mappings              Export sync.json mappings separately in their own directory.
  --sa-id <sa-id>                      Service account id.
  --sa-jwk-file <file>                 File containing the JSON Web Key (JWK) associated with the the service account.
  --verbose                            Verbose output during command execution. If specified, may or may not produce additional output.

Environment Variables:
  FRODO_HOST: Access Management base URL. Overrides 'host' argument.
  FRODO_REALM: Realm. Overrides 'realm' argument.
  FRODO_USERNAME: Username. Overrides 'username' argument.
  FRODO_PASSWORD: Password. Overrides 'password' argument.
  FRODO_SA_ID: Service account uuid. Overrides '--sa-id' option.
  FRODO_SA_JWK: Service account JWK. Overrides '--sa-jwk-file' option but takes the actual JWK as a value, not a file name.
  FRODO_NO_CACHE: Disable token cache. Same as '--no-cache' option.
  FRODO_TOKEN_CACHE_PATH: Use this token cache file instead of '~/.frodo/TokenCache.json'.
  FRODO_CONNECTION_PROFILES_PATH: Use this connection profiles file instead of '~/.frodo/Connections.json'.
  FRODO_AUTHENTICATION_SERVICE: Name of a login journey to use.
  FRODO_DEBUG: Set to any value to enable debug output. Same as '--debug'.
  FRODO_MASTER_KEY_PATH: Use this master key file instead of '~/.frodo/masterkey.key' file.
  FRODO_MASTER_KEY: Use this master key instead of what's in '~/.frodo/masterkey.key'. Takes precedence over FRODO_MASTER_KEY_PATH.

Some commands include samples in the help, others don't, the idm commands currently don't. Frodo's support for idm config export and import is extensive/complete, but the commands to do so are not very granular. Basically, the idm commands export/import EVERYTHING in idm. In the future we will be adding individual commands for individual config types like we do for most of the configuration in AM but for now that is what it is (notable exception is the frodo mapping command, which we added recently to support the export and import of idm mappings via its own command).

I recommend you do this to export everything into a directory ./idm:

% frodo idm export -AD idm volker-demo
Connected to https://openam-volker-demo.forgeblocks.com/am [alpha] as service account Frodo-SA-1700696937262 [dc837f84-3643-4c5d-9c05-acb73c7d02f4]
[========================================] 100% | 67/67 | Exported 67 config entities.
%

That will dump everything into the idm sub-directory inside your current directory:

% ll idm
total 688
drwxr-xr-x  30 vscheuber  staff   960B Jul 30 16:57 .
drwxr-xr-x   3 vscheuber  staff    96B Jul 30 16:57 ..
-rw-r--r--   1 vscheuber  staff   8.7K Jul 30 16:57 access.json
-rw-r--r--   1 vscheuber  staff    16K Jul 30 16:57 alphaOrgPrivileges.json
drwxr-xr-x   3 vscheuber  staff    96B Jul 30 16:57 appTemplate
-rw-r--r--   1 vscheuber  staff   3.6K Jul 30 16:57 audit.json
-rw-r--r--   1 vscheuber  staff   1.3K Jul 30 16:57 authentication.json
-rw-r--r--   1 vscheuber  staff    16K Jul 30 16:57 bravoOrgPrivileges.json
drwxr-xr-x  18 vscheuber  staff   576B Jul 30 16:57 emailTemplate
drwxr-xr-x   6 vscheuber  staff   192B Jul 30 16:57 endpoint
-rw-r--r--   1 vscheuber  staff   434B Jul 30 16:57 external.email.json
-rw-r--r--   1 vscheuber  staff   441B Jul 30 16:57 external.emailDefault.json
drwxr-xr-x   4 vscheuber  staff   128B Jul 30 16:57 fieldPolicy
-rw-r--r--   1 vscheuber  staff   644B Jul 30 16:57 internal.json
-rw-r--r--   1 vscheuber  staff   195K Jul 30 16:57 managed.json
drwxr-xr-x  10 vscheuber  staff   320B Jul 30 16:57 mapping
-rw-r--r--   1 vscheuber  staff    66B Jul 30 16:57 policy.json
-rw-r--r--   1 vscheuber  staff   678B Jul 30 16:57 privilegeAssignments.json
-rw-r--r--   1 vscheuber  staff    46B Jul 30 16:57 privileges.json
drwxr-xr-x   6 vscheuber  staff   192B Jul 30 16:57 provisioner.openicf
-rw-r--r--   1 vscheuber  staff   406B Jul 30 16:57 provisioner.openicf.connectorinfoprovider.json
-rw-r--r--   1 vscheuber  staff    42K Jul 30 16:57 repo.ds.json
-rw-r--r--   1 vscheuber  staff    39B Jul 30 16:57 router.json
-rw-r--r--   1 vscheuber  staff   3.1K Jul 30 16:57 secrets.json
-rw-r--r--   1 vscheuber  staff   206B Jul 30 16:57 selfservice.kba.json
-rw-r--r--   1 vscheuber  staff   823B Jul 30 16:57 selfservice.terms.json
drwxr-xr-x   5 vscheuber  staff   160B Jul 30 16:57 servletfilter
-rw-r--r--   1 vscheuber  staff   7.3K Jul 30 16:57 sync.json
drwxr-xr-x   7 vscheuber  staff   224B Jul 30 16:57 ui
drwxr-xr-x   6 vscheuber  staff   192B Jul 30 16:57 ui.context
%

This is basically a complete dump of all idm configuration into individual files and sub-directories. You can either import all of it using the equivalent import command, or you can delete everything you do not want to modify, modify what's left, and run the equivalent import command, which will then import what's left. Import command:

frodo idm import -AD idm volker-demo

The following commands support idm export import in some fashion:

command	description
frodo idm	discussed above - export/import all idm config
frodo mapping	export/import idm mappings
frodo app	export/import new application objects, includes both AM and IDM configuration making up the application
frodo config	export/import full env configuration. great for backup/restore of whole platform or cloud environments

Let me know if I can help with more info and stay tuned for the fine-grained IDM commands in 2.x versions.

debasisdwivedy commented 1 month ago

Hi,

Thanks for the response.

We have an on-premise IDM installation. The IDM export command is as below:

frodo idm export [options] [host] [realm] [username] [password]

In case of IDM, these options does not make sense. What is a 'realm' in context of IDM?

I tried running the below command for our IDM installation:

frodo idm export -AD idm https://<HOSTNAME>/admin / openidm-admin openidm-admin

Error getting tokens HTTP client error Code: ERR_BAD_REQUEST Status: 404 Unrecognized combination of options or no options...

Is there a way to export config from on-premise IDM installation using FRODO.

Regards,

Debasis

vscheuber commented 1 month ago

@debasisdwivedy if you are running a local IDM instance without AM (not a platform install, not ID Cloud) then you don't really need a tool to backup your config, as IDM leverages already a file-based configuration, so all you have to do is zip up your idm project folder and you will have something that looks very similar (not equal but similar) to what frodo idm export -AD idm <env host url> produces.

We have not had any requests to support standalone IDM for that reason. What is your scenario that makes the file-backup unfeasible? We are happy to entertain supporting standalone-IDM if there is a valid use case for it.

debasisdwivedy commented 1 month ago

Hi,

Our use case is as below:

We want to export an on-premise IDM configuration and import it into ID Cloud.

Is there a way with FRODO to achieve the same.

Regards, Debasis

vscheuber commented 3 weeks ago

Hi @debasisdwivedy, unfortunately not today. I think you bring up a valid use case, though.

debasisdwivedy commented 3 weeks ago

Hi @vscheuber ,

One more question i have, if we install platform UI in our on-premise , can we use FRODO to export IDM config for the on-premise environment and import it into ID cloud and visa versa.

Is it supported?

Regards,

vscheuber commented 3 weeks ago

Hi @debasisdwivedy , frodo expects a platform deployments, meaning it expects AM to be the OP (OpenID Connect Provider) for IDM. Frodo simply doesn't support the IDM-internal authentication, currently.

Funny enough, the platform UI has the same requirements as frodo: it requires a platform deployment (hence the name platform UI). So in order to run the platform UI you must install AM and configure it as the OP for IDM. That will make both the platform UI and frodo work.

For frodo, the platform UI is not a prerequisite, though. So as long as you install AM and configure it according to the Ping platform guide, frodo will work.

debasisdwivedy commented 3 weeks ago

Hi @vscheuber ,

One question related to platform UI local deployment export and import of IDM:

As per the platform set-up guide : https://backstage.forgerock.com/docs/platform/7/platform-setup-guide/

The URL's are as below: `Setting AM URL as http://amdc1.midships.io:8080/am

Setting AM ADMIN URL as http://amdc1.midships.io:8080/am/ui-admin

Setting ANALYTICS API URL as

Setting IDM REST URL as http://idmdc1.midships.io:8081/openidm

Setting IDM ADMIN URL as http://idmdc1.midships.io:8081/admin

Setting IDM UPLOAD URL as http://idmdc1.midships.io:8081/upload

Setting IDM EXPORT URL as http://idmdc1.midships.io:8081/export

Setting ENDUSER URL as http://platformuidc1.midships.io:8082/enduser

Setting PLATFORM ADMIN URL as http://platformuidc1.midships.io:8082/platform

`

How to export and import IDM config using FRODO for local platform deployment. The export mechanism for IDCloud and platform UI seems to be different i.e i cannot use the command below like i do for ID cloud

frodo idm export -AD idm https://openam-volker-demo.forgeblocks.com/am /alpha < USERNAME > < PASSWORD >

Regards, Debasis

debasisdwivedy commented 3 weeks ago

@vscheuber Just for context we have done a platform deployment and we are not able to export/import using frodo. The error is as below:

frodo idm export -AD idm http://amdc1.midships.io:8080/am / < USERNAME > < PASSWORD > A new version of frodo is available. Please run 'frodo -v' for more details. Error getting tokens Unsupported deployment type 'classic' Unrecognized combination of options or no options...

Usage: frodo idm export [options] [host] [realm] [username] [password]

Does frodo expect the Oath client with a particular name , for example in ID Cloud the client name is "RCSClient" whereas in our deployment the name is "idm-provisioning"

We tried with all the url's above and got the same error.

Could you give us an example command how you run FRODO with local platform deployment, that would be helpful.

Regards, Debasis

vscheuber commented 3 weeks ago

Hi @debasisdwivedy , thank you for your patience!

You are right, Frodo does make assumptions about the oauth client id. It uses the client the Admin UI typically uses. It differs between identity cloud and a typical forgeops (platform) deployment:

identity cloud: idmAdminClient forgeops: idm-admin-ui

For your use case, you want your deployment to look like a forgeops deployment. If you can check if you have an idm-admin-ui client, you should already be set. If not, create one that allows the authorization code grant and is pretty much a copy of your existing client just with the name Frodo would expect.

Next you may have to tell Frodo what your deployment type is (even though it should identify your deployment as forgeops once you create a idm-admin-ui client in your root realm). In case it doesn't, use -t forgeops to override the autodetected type.

So one of these two options should work for you, once you have a idm-admin-ui client:

frodo idm export -AD idm http://amdc1.midships.io:8080/am / < USERNAME > < PASSWORD >

or

frodo idm export -AD idm -t forgeops http://amdc1.midships.io:8080/am / < USERNAME > < PASSWORD >

Based on our conversation here, I am thinking we should allow you to override the oauth2 client id to make this easier. Let's first confirm that with these steps you can successfully access the env and export IDM data. Once that is established, we can discuss how to make this easier and more intuitive.

vscheuber commented 3 weeks ago

@debasisdwivedy a quicker way to test if frodo can successfully connect is to use the info command:

frodo info -t forgeops http://amdc1.midships.io:8080/am < USERNAME > < PASSWORD >

vscheuber commented 2 weeks ago

@debasisdwivedy check out Frodo CLI 2.0.6-0 and see if that makes your process easier:

Added

Improve support for custom platform deployments (non-forgeops or customized forgeops)
- #429: Added options to support custom oauth2 clients used to obtain the access token for IDM API calls:
  - --login-client-id <client-id> Specify a custom OAuth2 client id to use a your own oauth2 client for IDM API calls in deployments of type "cloud" or "forgeops". Your custom client must be configured as a public client and allow the authorization code grant using the "openid fr:idm:*" scope. Use the "--redirect-uri" parameter if you have configured a custom redirect uri (default: "/platform/appAuthHelperRedirect.html").
  - --login-redirect-uri <redirect-uri> Specify a custom redirect URI to use with your custom OAuth2 client (efault: "/platform/appAuthHelperRedirect.html").
  The above options can also be supplied through environment variables:
  - FRODO_LOGIN_CLIENT_ID OAuth2 client id for IDM API calls. Overridden by '--login-client-id' option.
  - FRODO_LOGIN_REDIRECT_URI Redirect Uri for custom OAuth2 client id. Overridden by '--login-redirect-uri' option.
- #359: Added an option to support custom IDM host URLs for all IDM API calls (e.g. platform deployments hosting AM and IDM on/in different DNS hosts/domains):
  - --idm-host <idm-host> IDM base URL, e.g.: https://cdk.idm.example.com/myidm. Use only if your IDM installation resides in a different domain and/or if the base path differs from the default "/openidm".
  The above option can also be supplied through an environment variable:
  - FRODO_IDM_HOST IDM base URL. Overridden by '--idm-host' option.
Note: All the above options are also persisted in connection profiles so they only have to specified once and after that they come out of the connection profile.

debasisdwivedy commented 1 week ago

Hi @vscheuber ,

Sorry for the late response i will try it by next week ( i am on leave and don't have access to my machine) and keep you posted.

Regards, Debasis

vscheuber commented 1 week ago

No worries, @debasisdwivedy , take your time! I hope it solves your use case, even if it is not a full support of IDM-only.

rockcarver / frodo-cli

Support IDM-only deployments #429

Added