improve scale performance

[micahstubbs]

improve scale performance

better handle concurrency for

• many puppeteer + headless chrome instances
• many network requests to GCP Storage to upload images to the specified bucket

[micahstubbs]

as a consequence of writing to the filesystem, we need a way of resizing pngs from the original dimensions to thumbnail dimensions

this looks useful, https://malcoded.com/posts/nodejs-image-resize-express-sharp

also http://sharp.dimens.io/en/stable/api-input/

[micahstubbs]

https://unix.stackexchange.com/questions/225401/how-to-see-full-log-from-systemctl-status-service

to see the logs from our screenshot-bot systemd process and follow their progress

journalctl -u screenshot-bot -f 

[micahstubbs]

to find a string mdcscry in filenames in a directory

tree -f | grep mdcscry

[micahstubbs]

nginx autoindex is pretty nifty

https://www.keycdn.com/support/nginx-directory-index

location /somedirectory/ {
    autoindex on;
}

check out the result at https://screenshot.micah.fyi/screenshots/

[micahstubbs]

ok, so wrapping up the main tactics to improve performance that worked are:

• decouple screenshotting from gcp bucket uploading. gcp bucket uploading in parallel caused problems - reset connections and rate-limiting
• switch from caolan/async concurrency to client-side concurrency (one small batch per API call)
• use a batch size of 4, down from the initial 12 block batch size

[micahstubbs]

ok, after doing this scale exercise and bulk processing ~28k blocks, I'm resizing the instance from

16 vCPU, 104 GB memory

to

1 vCPU, 6 GB memory

[micahstubbs]

some conversation from the d3 slack

text

Curran Kelleher [4:53 AM] Very cool! Did you ever resolve that memory leak issue? Micah Stubbs [1:43 PM] good question - I found a workaround my suspicion is that (edited) ```await page.close() await browser.close()``` (edited) somehow isn't enough to fully close a pupetteer managed headless chrome instance. a proper solution would involve figured out how to close/end/kill those puppeteer + headless chrome processes that get spun up every time the `screenshot.js` script is runs. a clever solution might involve specifying the desired concurrency up front (in my case I experimented with concurrencies of 10, 12, 16, 8, 4, before landing on concurrency 4) and then starting puppeteer headless chrome processes for each desired level of js event loop concurrency (4 in my case) then re-using those same process so long as there were more pages left in the queue to screenshot (for what it's worth, my unsatisfying workaround was to use a server with 104gb of RAM, then manually restart the server when the memory was almost full. )

https://d3js.slack.com/archives/C0LBA67MG/p1541246020029600