UserId migrator is not migrating the JDBC secondary user stores

[LahiruLS]

During the migration to the WSO2 IS 5.10.0 we have the component for user id migration. In the user-id migration, we can migrate the secondary user stores as well. But it was noticed that the migration client does not migrate the secondary user stores even though the necessary configurations are set.

After debugging the migration-client, we identified that when migrating the secondary user stores, the client will get the user store managers from the UserRealm[1]. But at the time of migration execution, the secondary user stores are not deployed. Therefore the user store manager relevant to the secondary user store is not retrieved. It throws the following exception and skips the migration.

TID: [-1234] [] [2020-08-12 19:34:21,593] [] ERROR {org.wso2.carbon.is.migration.service.v5100.migrator.UserIDMigrator} - Invalid domain name secondaryUS provided. No user store found for the given domain name.
TID: [-1234] [] [2020-08-12 19:34:21,599] [] ERROR {org.wso2.carbon.is.migration.MigrationClientImpl} - Migration process was stopped. org.wso2.carbon.identity.core.migrate.MigrationClientException: Invalid domain name provided. No user store found.
at org.wso2.carbon.is.migration.service.v5100.migrator.UserIDMigrator.migrate(UserIDMigrator.java:185)
at org.wso2.carbon.is.migration.VersionMigration.migrate(VersionMigration.java:52)
at org.wso2.carbon.is.migration.MigrationClientImpl.execute(MigrationClientImpl.java:85)
at org.wso2.carbon.identity.core.internal.IdentityCoreServiceComponent.activate(IdentityCoreServiceComponent.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(ServiceComponent.java:260)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.activate(ServiceComponentProp.java:146)
at org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(ServiceComponentProp.java:345)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(InstanceProcess.java:620)
at org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(InstanceProcess.java:197)
at org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:343)
at org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.java:222)
at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:113)
at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:985)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:866)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:525)
at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:544)
at org.wso2.carbon.core.init.CarbonServerManager.initializeCarbon(CarbonServerManager.java:529)
at org.wso2.carbon.core.init.CarbonServerManager.removePendingItem(CarbonServerManager.java:305)
at org.wso2.carbon.core.init.PreAxis2ConfigItemListener.bundleChanged(PreAxis2ConfigItemListener.java:118)
at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:973)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)

We need to fix this and all other places where we use similar logic in the migration client apart from the UserIDMigrator.

IMO it is better to migrate only the PRIMARY user store during the migration with an option and migrate the other secondary user stores as a post-migration task via some external tool etc. The reasons are,

1. Performance issues during the migration process when the user base is large
2. If we need to migrate the secondary user stores during the migration then we have to forcefully deploy the secondary user stores and do it. IMO that is not a good approach.

[1] https://github.com/wso2-extensions/identity-migration-resources/blob/master/components/org.wso2.is.migration/migration-service/src/main/java/org/wso2/carbon/is/migration/service/v5100/migrator/UserIDMigrator.java#L178

[DMHP]

https://github.com/wso2/product-is/issues/10584

[DMHP]

The following approach is discussed to solve this issue:

• Secondary Userstore migration and tenant migration are considered as independent migrations from product migration.
• After the product migration, the traffic related to the userstores and tenants should be routed to a new table/tables.
• The data in the old table/tables should be migrated to the new table/tables gradually. (migrate the users and tenants as chunks)
• After the migration, the old table/tables can be removed and the new table/tables can be used permanently.

The advantages of this approach are :

• Can handle the performance issues in the migration process by migrating a large user base.
• Can do the migration without downtime.

[ruwanta]

It would be better if we can do tenant migration without user-store migration. The reason is that, it is presently difficult to do traffic routing based on tenant, as tenant qualified URL support is being added on recent versions only.

There will be two steps.

1. Product migration with all the tenants migrated (only configs are migrated)
2. Secondary user store data migration, tenant by tenant
3. Re-configure user-store to new user store, tenant by tenant.

[DMHP]

https://github.com/wso2/product-is/issues/10524

[gomathyk]

https://is.docs.wso2.com/en/latest/setup/migrating-userstore-managers/