Make HTTP request (1)

[sarah-tully]

• Choose http library:
  • Request
  • beautiful soup
  • scrapy (or whatever other libraries you find)
• write code to hit senate FR page
• print output (response/body) to console/standard output

[wickkidd]

Not a bad read on scraping libraries: https://elitedatascience.com/python-web-scraping-libraries

[sarah-tully]

Overall project goal:

scrape https://www.foreign.senate.gov/hearings/ for documents uploaded as “related files” in business meetings.

Assignment objective:

choose a library/framework[https://www.quora.com/What-is-the-difference-between-a-library-framework-and-a-language] that would “best” serve our goal

Define “best”:

1. Uses python
2. Is appropriate for building a focused crawler
3. Can handle redirects
4. Is appropriate for beginners

Findings:

Not gonna work:

• Apache Nutch, Heritrix, WebSphinix, JSPider, GNUWGet, Wire, Pavuk, Teleport, WebHTTrack are all scrapers that could get the job done but don’t use python, so they’re not good candidates for our purpose.

Will work:

• Requests + lxml -- (this would work but requires familiarity with xpath or css – less accessible than beautiful soup but would work for parsing)
• Requests + Beautiful Soup 4 (Requests to make the http request, beautiful soup to parse it)
• Scrapy (it’s easy to use, built-in support for exporting to CSV, JSON, XML, it allows for scheduled scrapings (“spiders”), good for redirection)
• selenium (good for when the content you’re trying to scrape is delivered through JS)

Requests documentation Selenium documentation Scrapy documentation

Examples:

Overview/use-case example for scrapy scraping with Requests and Beautiful Soup

Other resources:

• • http://www.aioptify.com/top-web-scraping-frameworks-and-librares.php
• • https://www.ijser.org/researchpaper/Comparison-of-Open-Source-Crawlers--A-Review.pdf (mostly section 3)
• • https://www.quora.com/What-are-the-best-python-libraries-for-web-scraping-and-web-crawling
• • https://www.quora.com/How-do-I-choose-between-using-Beautiful-Soup-or-Scrapy-It-seems-using-frameworks-like-Scrapy-could-address-many-issues-like-IP-blocking-and-data-storing-easier-than-using-a-pure-parsing-library-like-Beautiful-Soup
• • http://www.discoversdk.com/blog/beautifulsoup4-vs-scrapy?utm_source=quora&utm_medium=article&utm_campaign=aaron

FINAL THOUGHTS:

there are lots and lots of ways to do this. Based on our goals, I think either Requests + Beautiful Soup or Scrapy would be best suited for our needs. Scrapy seems to be more the more bougie option - less bare-boned than Requests + BS4. I think for a beginning project like this, perhaps using a less built out method than scrapy would be useful, so we can write our own code in response to problems, rather than finding a pre-made answer that we might find with Scrapy.