This repository hosts the course website of Tilburg University's open education class on "Online Data Collection and Management" (oDCM) - learn how to collect web data for your empirical research projects!
Learning goal of podcasts:
(1) How is web scraping used by companies?
business cases:, skyscanner for flight prices, inprijsverhoogd for market research
how do they setup their infrastructure: how often do they scrape, how do they store their data
ever had legal concerns or asked for permission?
what's the future of scraping, in your opinion?
(2) How are APIs used by firms
disclose data, functionality, or algorithms (contacts w/ google cloud); chartmetric API
what does your business do, exactly?
is API your core activity, or just a side activity? (powering the business, vs. being part of the actual business model)
how many customers do you have? and what type of customers?
what's the payment model you're using? why have you opted for that, versus another one? (e.g., free tiers, free developer plan, etc).
how do you decide on API retrieval thresholds?
what's your backend look like - all self-coded, versus some kind of provider?
OTHER QUESTIONS?
@RoyKlaasseBos: in the spirit of Hilke's comments: maybe such podcasts or video discussions exist already? Can you go look for them?
specific follow-ups to discuss with @RoyKlaasseBos:
Conduct interview with business (e.g., market research firm) on (a) how they have setup their infrastructure (e.g., how often, storage, legal concerns/permission); + FUTURE of web scraping business
was originally posted as #14