NicholasBoll / github-migration

Migrate github issues, pull requests and comments from any github source to any other github target. Works with history rewriting.
38 stars 14 forks source link

connection timeout on creating issues #15

Open mkumari123 opened 3 years ago

mkumari123 commented 3 years ago

Any suggestion on why do I get this error after every 5 Pull request is created on create issues and create comments and how to resolve this: /github-migration/node_modules/aws-sdk/lib/config.js:478 if (err) throw err; ^

TimeoutError: EC2 Metadata roleName request returned error at Timeout.connectTimeout [as _onTimeout] (/github-migration/node_modules/aws-sdk/lib/http/node.js:69:15) at ontimeout (timers.js:436:11) at tryOnTimeout (timers.js:300:5) at listOnTimeout (timers.js:263:5) at Timer.processTimers (timers.js:223:10) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! migrate-github@1.0.0 createIssues: node ./createIssues.js npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the migrate-github@1.0.0 createIssues script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:

phillipbroberts commented 3 years ago

We are getting this issue as well. Exactly.

xuatz commented 3 years ago

From what I can see the only thing that is calling aws from createIssues.js seems to be createIssue() which calls processImages() which uploads images to s3. Maybe there are too many images in the issue body processImages(issue.body) and you are hitting the rate limit for aws calls or s3 uploads.

Maybe try to change Promise.all in the function below into something like bluebird Promise.mapSeries or a native sequential implementation, and add a delay/sleep after each execution.

// processImages.js
const processImages = async (content) => {
  const imgRegExp = /!\[([^\]]+)\]\(([^\)]+)\)/
  const imgMatchAll = new RegExp(imgRegExp, 'g')

  if (config.s3Bucket) {
    return Promise.all(
      (content.match(imgMatchAll) || [])
        .map(img => {
          const [_, title, oldUrl] = img.match(imgRegExp)
          return request({
            method: 'GET',
            encoding: null, // force a buffer
            url: oldUrl,
            transform: (body, response) => ({
              headers: response.headers,
              body,
            })
          })
            // .then(console.log)
            // .then(() => { process.exit(1) })
            .then(response => uploadImage(config.s3Bucket, response.headers['content-type'])(response.body))
            .then(newUrl => {
              console.log('Uploaded image: ', newUrl)
              return { oldUrl, newUrl }
            })

Maybe that would fix the problem, after all, you managed to run the script for 5 pull requests as you mentioned.