Dockerization - Live web scraping

[N2ITN]

Status

Assigning to @N2ITN Please use this branch https://github.com/N2ITN/are-you-fake-news/tree/develop-dockerize

Issue

There are several webscraping functions that were formerly on AWS lambda that need to be moved into one service.

There are 3 different sets of functionality here. 1) spidering a news site for article URLs. 2) Scraping a single URL for article text. 3) Calling functionality 2 for a list of URLs using asyncio. Currently located in ./_scrape_lambda/code/.

Tasks

• [x] Add all of these services to a dockerfile under ./docker/web_scraper
• [ ] All services should be wrapped in a simple flask API.
• [ ] This service should be able to integrate nicely with the existing ./docker/web/ service
• [ ] The service should be fast asynchronous and not exceed 100 requests/second to avoid potential ISP flagging.
• [ ] Add unit tests

[N2ITN]

Back burnering this for now as replacing the existing AWS Lambda calls potentially creates more problems that it solves.

The challenges introduced by adding this functionality to the docker cluster include:

• Concurrency - getting Asyncio to call Asyncio is hard
• Speed - AWS has faster internet than most people and IO is the bottleneck here
• ISP flagging - too many outgoing requests can get someone temporarily banned by their ISP. This way just makes one call to an AWS Lambda that returns many results.
• IP bans - websites might end up blocking a user's IP for excessive requests (though we try to be gentle on their websites). Lambda IPs are multitudinous and can't all be blocked.

The problems that implementing this issue would solve:

• Separation from a central AWS account - no external dependencies
• Flexibility - the webscraper code could be changed by users
• Potential cost overruns - Excessive use could rack up bills on my account. The Lambda APIs can handle several millions of requests before incurring any real costs, but it is possible.

The challenges could be overcome and it would still be worth having this as an option. Unless there is someone who feels passionately about working on this issue, I will pause work on it until the other parts of the app are implemented in docker-compose.