N2ITN
commented
6 years ago Closing this, as I moved a lot of compute to Lambda, the only thing left is to move the MongoDB away from the web server, which should be its own Issue.
Closed N2ITN closed 6 years ago
N2ITN
commented
6 years ago Closing this, as I moved a lot of compute to Lambda, the only thing left is to move the MongoDB away from the web server, which should be its own Issue.
Status
Assigning to @N2ITN, anyone who can help!
Issue
Right now the website might act up under increased load, due to both slow API requests (scraping, plotting etc on AWS Lambda instances) and to the CPU associated with querying the local MongoDB and serving the web page.
Right now my solution is using asyncio with gunicorn on a server with 4 cores, and asyncio on the Lambda that calls the webscraping. I'm also only scraping webpages that havn't been scraped in 4 hours.
Tasks