Include tips/lessons learnt from other instructors in own content

[hannesdatta]

Podcasts

Here are the highlights of a podcast about web scraping by a bootcamp instructor that touches on a variety of topics related to oDCM (navigating the DOM, cleaning text data, timers, selenium vs other tools)

• Kimberly Fessel (PhD) - Metis
• Request and Beautifulsoup
• Two strategies:
  1. Look for unique attributes (ids / classes)
  2. Navigate the Document Object Model - DOM (children, sibling) ~ tree like structure
• Good practise website:
  • Boxofficemojo
    • Well-structured
    • Movie revenue, publish date, actors
  • Sports Reference
    • Player level stats (baseball, basketball, football, hockey)
  • Wikipedia
    • But keep in mind the maximum scraping limit (1 page per second)
    • Tutorial
• Importance of including pauses in your request (to avoid getting blocked)
  • It could be blocked for a hour, a day, but also indefinitely
• Importance of saving the data you have collected (site structure may change over time)
  • Write as a csv or store as pickle files
• Start out in Jupyter Notebooks to make sure you have the right syntax to get the data → convert to a Python script → set-up scheduling
• Stripping out characters ("$", ",", non-printing characters)
  • Importance of regular expressions (start out with replace() initially
  • Convert dates and times to pandas timeseries
• Limitations of BeautifulSoup and request
  • Does not work for sites that are dynamically loading content (hitting a database and pulling in information).
  • Mostly JavaScript websites (YouTube, Open Table)
  • Selenium is the solution; launches a Google Chrome driver; sometimes it as simple as launching the site with selenium and then processing the data with request and Beautifulsoup.
  • Other advantages: clicking on things and filling out fields
  • Scrapy - cloud deployment and built a "spider" (scraper that keeps on going and look for new links)
  • Importance of visualising your results dynamically/interactively (D3, Plotly, Tableau)
  • Data widgets getting more mainstream (e.g., NYT) - people getting more data literate

Legality of Web Scraping

Video that answers the question whether web scraping is legal. They share your view and illustrate this with recent law cases:

• Have to make a clear distinction between types of data:
  • Publicly available data (e.g., public LinkedIn profile)
  • User had made the data public
  • No account required for access
  • Not blocked by robots.txt
• hiQ Labs case (scraped public LinkedIn profiles for workplace analytics)
• Craigslist case (start-ups use their location data)
• Why scraping publicly available information online isn't a crime

Originally posted by @RoyKlaasseBos in https://github.com/hannesdatta/course-odcm/issues/14#issuecomment-733671043

[hannesdatta]

I've added the info at various places at the site.