Inspiration

[hannesdatta]

• [ ] @RoyKlaasseBos, which firms/contact persons for podcast?
• [ ]

[hannesdatta]

Learning goal of podcasts: (1) How is web scraping used by companies?

• business cases:, skyscanner for flight prices, inprijsverhoogd for market research
• how do they setup their infrastructure: how often do they scrape, how do they store their data
• ever had legal concerns or asked for permission?
• what's the future of scraping, in your opinion?

(2) How are APIs used by firms

• disclose data, functionality, or algorithms (contacts w/ google cloud); chartmetric API
• what does your business do, exactly?
• is API your core activity, or just a side activity? (powering the business, vs. being part of the actual business model)
• how many customers do you have? and what type of customers?
• what's the payment model you're using? why have you opted for that, versus another one? (e.g., free tiers, free developer plan, etc).
• how do you decide on API retrieval thresholds?
• what's your backend look like - all self-coded, versus some kind of provider?
• OTHER QUESTIONS?

@RoyKlaasseBos: in the spirit of Hilke's comments: maybe such podcasts or video discussions exist already? Can you go look for them?

[hannesdatta]

...something to kickstart: https://www.youtube.com/watch?v=2XfA0e4Bzkk parsehub may be relevant. we could use their videos also elsewhere in this course (e.g., on legal perspectives).

[RoyKlaasseBos]

Podcasts

Here are the highlights of a podcast about web scraping by a bootcamp instructor that touches on a variety of topics related to oDCM (navigating the DOM, cleaning text data, timers, selenium vs other tools)

• Kimberly Fessel (PhD) - Metis
• Request and Beautifulsoup
• Two strategies:
  1. Look for unique attributes (ids / classes)
  2. Navigate the Document Object Model - DOM (children, sibling) ~ tree like structure
• Good practise website:
  • Boxofficemojo
    • Well-structured
    • Movie revenue, publish date, actors
  • Sports Reference
    • Player level stats (baseball, basketball, football, hockey)
  • Wikipedia
    • But keep in mind the maximum scraping limit (1 page per second)
    • Tutorial
• Importance of including pauses in your request (to avoid getting blocked)
  • It could be blocked for a hour, a day, but also indefinitely
• Importance of saving the data you have collected (site structure may change over time)
  • Write as a csv or store as pickle files
• Start out in Jupyter Notebooks to make sure you have the right syntax to get the data → convert to a Python script → set-up scheduling
• Stripping out characters ("$", ",", non-printing characters)
  • Importance of regular expressions (start out with replace() initially
  • Convert dates and times to pandas timeseries
• Limitations of BeautifulSoup and request
  • Does not work for sites that are dynamically loading content (hitting a database and pulling in information).
  • Mostly JavaScript websites (YouTube, Open Table)
  • Selenium is the solution; launches a Google Chrome driver; sometimes it as simple as launching the site with selenium and then processing the data with request and Beautifulsoup.
  • Other advantages: clicking on things and filling out fields
  • Scrapy - cloud deployment and built a "spider" (scraper that keeps on going and look for new links)
  • Importance of visualising your results dynamically/interactively (D3, Plotly, Tableau)
  • Data widgets getting more mainstream (e.g., NYT) - people getting more data literate

Legality of Web Scraping

Video that answers the question whether web scraping is legal. They share your view and illustrate this with recent law cases:

• Have to make a clear distinction between types of data:
  • Publicly available data (e.g., public LinkedIn profile)
  • User had made the data public
  • No account required for access
  • Not blocked by robots.txt
• hiQ Labs case (scraped public LinkedIn profiles for workplace analytics)
• Craigslist case (start-ups use their location data)
• Why scraping publicly available information online isn't a crime