micahstubbs / screenshot-service

services to create a screenshot of a web page. optimized for screenshotting interactive data graphics.
0 stars 0 forks source link

handle multiple simultaneous requests gracefully #21

Closed micahstubbs closed 5 years ago

micahstubbs commented 5 years ago

this doesn't look right - 10 different requests, but writing to the bucket 10 times with the same filename of the first request url 🤔

screen shot 2018-10-29 at 12 24 14 am

micahstubbs commented 5 years ago

this is helpful https://codeforgeek.com/2014/09/handle-get-post-request-express-4/

micahstubbs commented 5 years ago

this answer recommending limiting concurrency with queues from the async package https://github.com/caolan/async

looks promising https://stackoverflow.com/a/9547175/1732222

var async = require("async");

function upload_file(file, callback) {
    // Do funky stuff with file
    callback();
}

var queue = async.queue(upload_file, 10); // Run ten simultaneous uploads

queue.drain = function() {
    console.log("All files are uploaded");
};

// Queue your files for upload
queue.push(files);

queue.concurrency = 20; // Increase to twenty simultaneous uploads
micahstubbs commented 5 years ago

ran into this https://www.cyberciti.biz/faq/linux-unix-bsd-nginx-413-request-entity-too-large/

micahstubbs commented 5 years ago

and then the related error with express https://stackoverflow.com/questions/19917401/error-request-entity-too-large

micahstubbs commented 5 years ago

this is useful to see the logs of the service

journalctl -u screenshot-bot.service -f