cofacts / rumors-api

GraphQL API server for clients like rumors-site and rumors-line-bot
https://api.cofacts.tw
MIT License
109 stars 26 forks source link

cleanupUrls errors on production cron job #210

Closed MrOrz closed 4 years ago

MrOrz commented 4 years ago

In cron.log:

100 / 1742783 Processed
Possibly Unhandled Rejection at: Promise  Promise {
  <rejected> TypeError: Cannot read property 'hits' of undefined
      at processUrls (/srv/www/build/scripts/cleanupUrls.js:77:26)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async main (/srv/www/build/scripts/cleanupUrls.js:176:3)
}  reason:  TypeError: Cannot read property 'hits' of undefined
    at processUrls (/srv/www/build/scripts/cleanupUrls.js:77:26)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async main (/srv/www/build/scripts/cleanupUrls.js:176:3)

build/scripts/cleanupUrls:

     69   while (processedCount < total) {
     70     const {
     71       hits,
     72       _scroll_id
     73     } = await _get__("client").scroll({
     74       scroll: '30s',
     75       scrollId
     76     });
     77     await processFn(hits.hits);
     78     processedCount += hits.hits.length;
     79     scrollId = _scroll_id; // eslint-disable-next-line no-console
     80
     81     console.info(`${processedCount} / ${total} Processed`);
     82   }

Seems that there are some error handling issue, or query does not return expected results.

It has been there for a few days, current disk level is at 30% full. Will need to mitigate this before all our disk space is eaten up by useless URLs.