search

apigeecs / apigee-migrate-tool

Export/Import Org Data. Import from CSV files
MIT License
52 stars 50 forks source link

All data is not getting imported #48

Open manojietpumca opened 4 years ago

manojietpumca commented 4 years ago

Hi, I have used "grunt exportAll -v" all data is not getting imported to target environment. So is there any singe command to import all the exported data to local drive.? image

Can all the data be imported using single command.?

Or we need to run all the below commands manually one by one: grunt importTargetServers grunt importProxies grunt importSharedFlows grunt importDevs grunt importProducts grunt importApps grunt importKeys grunt importProxyKVM grunt importEnvKVM grunt importOrgKVM

kurtkanaskie commented 4 years ago

You can use "grunt importAll -v", see the Gruntfile.js for all the commands. Also, see the output from "grunt -v".

If you see issues, let us know.

manojietpumca commented 4 years ago

I used command "grunt importAll -v" then also it is not importing any thing while I have not change anything in Gruntfile.js. Is anything required to change in Gruntfile.js?

Below is dummy sample of my config.json:

module.exports = {

from: {
    version: '19.01',
    url: 'http://keng03-dev01-api-mky23-app-ilb-1111111111.int.dev.myabc.com',
    userid: 'mky@abc.com',
    passwd: 'ykm!',
    org: 'abc',
    env: 'test'
},
to: {
    version: '19.01',
    url: 'http://keng03-dev01-api-kkk17-app-ilb-2222222222.int.dev.myabc.com',
    userid: 'mky@abc.com',
    passwd: 'ykm!',
    org: 'abc',
    env: 'test'
}

} ; **After running command "grunt -v" I found below lines in command prompt:**

C:\Users\manoj.yadava\apigee-migrate-tool>grunt -v Initializing Command-line options: --verbose, --gruntfile=C:\Users\manoj.yadava\apigee-migrate-tool\Gruntfile.js

Reading "Gruntfile.js" Gruntfile...OK

Registering Gruntfile tasks. Reading package.json...OK Parsing package.json...OK Initializing config...OK

Registering "grunt-available-tasks" local Npm module tasks. Reading C:\Users\manoj.yadava\apigee-migrate-tool\node_modules\grunt-available-tasks\package.json...OK Parsing C:\Users\manoj.yadava\apigee-migrate-tool\node_modules\grunt-available-tasks\package.json...OK Loading "available_tasks.js" tasks...OK

Registering "grunt-mkdir" local Npm module tasks. Reading C:\Users\manoj.yadava\apigee-migrate-tool\node_modules\grunt-mkdir\package.json...OK Parsing C:\Users\manoj.yadava\apigee-migrate-tool\node_modules\grunt-mkdir\package.json...OK Loading "mkdir.js" tasks...OK

Registering "tasks" tasks. Loading "app.js" tasks...OK

No tasks specified, running default tasks. Running tasks: default

Running "default" task

Running "availabletasks" task

Running "availabletasks:tasks" (availabletasks) task Verifying property availabletasks.tasks exists in config...OK File: [no files] Options: filter="exclude", tasks=["mkdir","availabletasks","warn","default"], sort, hideUngrouped=false, groups={}, descriptions={}, showTasks=["single","multi","user"], reporter="default" deleteAll => Alias for "warn", "deleteApps", "deleteDevs", "deleteProducts", "deleteProxies", "deleteFlowHooks", "deleteSharedFlows", "deleteTargetServers", "deleteEnvKVM", "deleteOrgKVM", "deleteProxyKVM", "deleteKeys", "deleteReports" tasks. deleteAllSpecs => Delete all Specs from abc [19.01] deleteApps => Delete all apps from org abc [19.01] deleteDevs => Delete all developers from org a [19bc.01] deleteEnvKVM => Delete all env-kvm from org abc environment test [19.01] deleteFlowHooks => Detach any configured flow hooks from org abc [19.01] deleteKeys => Delete all app keys from org abc [19.01] deleteOrgKVM => Delete all org-kvm from org a [bc19.01] deleteProducts => Delete all products from org abc [19.01] deleteProxies => Delete all proxies from org abc [19.01] deleteProxyKVM => Delete all proxy-kvm from org abc [19.01] deleteReports => Delete all reports from org abc [19.01] deleteSharedFlows => Delete all shared flows from org abc [19.01] deleteTargetServers => Delete all target servers from org abc [19.01] deployProxies => Deploy revision 1 on all proxies for org abc [19.01] deploySharedFlows => Deploy revision 1 on all shared flows for org abc [19.01] exportAll => Alias for "exportProxies", "exportProducts", "exportDevs", "exportSharedFlows", "exportApps", "exportFlowHooks", "exportTargetServers", "exportOrgKVM", "exportEnvKVM", "exportProxyKVM", "exportReports" tasks. exportAllSpecs => Export all Specs from org abc [19.01] exportApps => Export all apps from org abc [19.01] exportDevs => Export all developers from org abc [19.01] exportEnvKVM => Export all env-kvm from org abc environment test [19.01] exportFlowHooks => Export all configured flow hooks from org abc [19.01] exportOrgKVM => Export all org-kvm from org abc [19.01] exportProducts => Export all products from org abc [19.01] exportProxies => Export all proxies from org abc [19.01] exportProxyKVM => Export all proxy-kvm from org abc [19.01] exportReports => Export all reports from org abc [19.01] exportSharedFlows => Export all shared flows from org abc [19.01] exportTargetServers => Export all target servers from org abc [19.01] importAll => Alias for "importTargetServers", "importProxies", "importProducts", "importDevs", "importApps", "importSharedFlows", "importFlowHooks", "importOrgKVM", "importEnvKVM", "importProxyKVM", "importKeys", "importReports" tasks. importAllSpecs => Import all Specs to abc [19.01] importApps => Import all apps to org abc [19.01] importDevs => Import all developers to org abc [19.01] importEnvKVM => Import all env-kvm to org abc environment test [19.01] importFlowHooks => Import all configured flow hooks to org abc [19.01] importKeys => Import all app keys to org abc [19.01] importOrgKVM => Import all org-kvm to org abc [19.01] importProducts => Import all products to org abc [19.01] importProxies => Import all proxies to org abc [19.01] importProxyKVM => Import all proxy-kvm to org abc [19.01] importReports => Import all reports to org abc [19.01] importSharedFlows => Import all shared flows to org abc [19.01] importTargetServers => Import all target servers to org abc [19.01] readCSVApps => Read CSV file for apps readCSVDevs => Read CSV file for devs tasks => Alias for "availabletasks" task. undeployProxies => UnDeploy revision 1 on all proxies for org abc [19.01] undeploySharedFlows => UnDeploy revision 1 on all shared flows for org abc [19.01]

Done.

Can you please correct me if missed anything. I am also not able to verify from the gateway.environment UI page.

kurtkanaskie commented 4 years ago

The config.js file contain a from and to org, in your's they are the same, they should be different, as the purpose of the tool is to export from one org and import to another. Export places data "from" the org/env in the data directory. Import uses the data directory as the source for the "to" org/env. I just ran a test and everything worked as expected.

manojietpumca commented 4 years ago

Actually we do this for blue/green gateway upgrade. so our requirement is to migrate data form one gateway to other gateway having same org/env but different gateways. In my config.js different from and to gateways urls are there Is there any issue to migrate in same org/env even though the gateways are different? From and To url would be the same? If they are the different then also org/env would be different?

kurtkanaskie commented 4 years ago

Ah I see, this is for on-prem.

The two actions are separate. Export uses the "from" config, import uses the "to" config. So it shouldn't matter what's in "from", the import is just looking at the "to" config.

I suspect there's something else wrong. Are you sure you have the correct credentials?

As a debugging step, you could create a "facade" proxy in one of your good gateway orgs that is just a pass through to the other gateway Edge API.

For example, I created an "edge-facade-v1" proxy with a basepath of "/edge-facade/v1". And a HTTPTargetConnection URL: https://api.enterprise.apigee.com/v1 You would have to change this to be the actual "gateway" URL for your target gateway Edge API.

Then in my "to" config I used: url: 'https://api-test.kurtkanaskie.net/edge-facade'

Run trace on your facade API to see what's happening with the requests going to the "to" config / target gateway org.

Hope that helps.

manojietpumca commented 4 years ago

I am sure credentials are correct. There is a different way to migrated data from one gateway to other gateway but taking a lot of time, so wanted to use this tool for fast migration. instead of using command "grunt importAll -v" I used command "grunt importProxies" then the few proxies were migrated not all but I think using separate commands for different data's is not good practice.

One more thing some time gives error "Fatal error: Cannot read property 'name' of undefined" or some times "Fatal error: read ECONNRESET" during "grunt exportAll -v" what can be the cause for this.? while "runt importAll -v" finished just in one second, is it OK? Any idea how to debug the issue.?

kurtkanaskie commented 3 years ago

Circling back around, I've noticed some issues using importAll.

  1. Sometimes with very large numbers of entities, I've seen errors, not sure exactly why, may be the case with exportAll. I'd suggest using the individual commands as a workaround (e.g. grunt exportProxies).
  2. On importAll, if there are no targetservers and thus no targetservers directory in data, then the tool exists permaturely. Again, use the individual commands in that case.

See the order in "all_import_tasks.sh"