search

apigeecs / apigee-migrate-tool

Export/Import Org Data. Import from CSV files
MIT License
52 stars 50 forks source link

[Enhancement] API rate limiting #21

Closed cmische closed 5 years ago

cmische commented 6 years ago

Hi! Thanks for maintaining this tool!

I'm currently trying to use this tool to migrate data between some corporate Apigee installations, I've ran into an issue I've mostly been able to address which is that most of the tasks here will fire off asynchronous javascript requests as fast as possible without waiting for a callback. This was a problem for me because I didn't want to install grunt and NPM on my production servers, and was forced to use this tool over an SSH tunnel; which then gets overwhelmed with requests and dies whenever I try to migrate ~6000 developers or ~7000 something keys.

Now for the most part (and having no javascript experience), I was able to substitute out the request library in each of the tasks for the request-promise library which returns a promise for the callback of the async function, and then use the recent javascript 'async/await' functionality to make those tasks execute synchronously. (Awaiting on any request to callback before proceeding with the loop) However I've found the key.js task quite tricky to get to execute synchronously and is what I'm stuck on now.

Anyway, what I'm getting at is a rate limiting feature would be nice!

cmische commented 6 years ago

I just found that the https://www.npmjs.com/package/bottleneck library can easily be included to accomplish this.

A reddit thread I stumbled accross explaining it for other dummys like myself; https://www.reddit.com/r/javascript/comments/82gipw/okay_seriously_i_need_help_with_throttling_async/

I just declared a limiter and wrapped async.series for tasks/key.js in a limiter.submit(async.series, ...) and it worked beautifully. I imagine it could potentially work the same throughout and be optional.