Scans appear stuck processing

[bbertucc]

A client sent about 1000 pages to the scan. Their URL: https://learn.wordpress.org/.

The scans completed at scan.equalify.app, but they're still stuck, scanning on the dashboard.

It appears this is an API problem because pinging the API shows the same data.

API key: 5a87034ae1f740d4b6dbb9e16d7cc4a9.

[heythisischris]

Found the issue- we were passing too much data in the Body JSON payload when calling equalify-api (Lambda) -> processScans (Path):

{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "RequestEntityTooLargeException: 635215 byte payload is too large for the Event invocation type (limit 262144 bytes)"
}

I reduced the payload by removing unnecessary duplicate data from the array (such as id, url, userId, and propertyId) and just including a series of jobIds (i.e. [123,124,125]).

Fix should be out soon.

[heythisischris]

Fix pushed to both staging/prod, the learn.wordpress.org scan now goes through:

I'm now batching GET requests to https://scan.equalify.app in groups of 100 JobIds to prevent overloading the scan server.

Here's the commit for reference: https://github.com/EqualifyEverything/equalify-api/commit/940383dfb4fff4e47e772a461956d5b0ad7678c7

[heythisischris]

We are now using Lambda as a poor man's ECS. 💸

https://github.com/EqualifyEverything/equalify-api/commit/325ea4998b6565d4eb0ec387b863249245dd95b0

Since our Lambda has a timeout of 5 minutes, we're running processScans for 2 minutes. If it hasn't completed in that timeframe, we recursively call processScans where we left off. We can do this up to 750 times, which is roughly 24 hours. This should be enough to handle ~4000 scan jobs.

Since we're using Lambda w/ ARM Node.js @ 512mb, a full 24 hour run will set us back $0.60.

Happy to provide more detailed calcs, but this is a good-enough quick rough estimate.

[bbertucc]

Cool. I couldn't test this because of #449

[heythisischris]

Updated our scan processing logic: https://github.com/EqualifyEverything/equalify-api/commit/a9516cde8dfce25bba824fdc959e49ff76d560fa

• We have a scheduled job called runEveryFifteenMinutes which runs every 15 minutes and checks for pending scans
• If there are any pending scans, we kick off a new Lambda run with processScans. This should last about 15 minutes- if it cannot complete it in this timeframe, we'll simply wait for it to be kicked off again via runEveryFifteenMinutes

Benefits of our new architecture

• We no longer have recursively invoked Lambda functions, which can be tricky to track/maintain/debug
• We'll always make sure any scans are taken care of
• Logs are predictable
• Each user and batch of scans gets their own dedicated Lambda to processing, ensuring everyone has the seem processing speed

[bbertucc]

@heythisischris do I need to restart the WP scans I am running?

[heythisischris]

@bbertucc No, you shouldn't need to restart- give the scan processor 15-30 min to catch up and process these old pending scans

[bbertucc]

Sweet @heythisischris. Staying tuned.

[bbertucc]

Unfortunately the scans are still stuck processing.. looks like a bunch completed since your last update though. The user is wp@equalify.app.

[bbertucc]

This is fixed