POC: Scraping theprotocol.it

[NowanIlfideme]

Create a proof-of-concept for scraping the website theprotocol.it from Python.

Here are some possible libraries to use:

• beautifulsoup4 which gives the HTML content result tree;
• scrapy which seems more like a "do x, then y" approach rather than just static HTML;
• Selenium which is a powerful, but probably overkill, web browser automation library (see also resource Selenium for Python)

I've worked with bs4 but scrapy seems like something to look at too.

You could put the POC code somewhere in the package or alongside it. Options:

1. Put it in src/josh/scrapers/theprotocol.py
2. Put it in poc/theprotocol.py and just create a first script that works.
3. Put it in a Jupyter notebook under nb/scrape_theprotocol.ipynb so you have outputs to share.

[NowanIlfideme]

Later on, we can create a command line josh scrape theprotocol (maybe with params). And then expand to a similar interface for other sites (though the scrapers will be different).

[NowanIlfideme]

First get something in dict formats, later we can work on more structured metadata. ;)