PaulMcInnis / JobFunnel

Scrape job websites into a single spreadsheet with no duplicates.
MIT License
1.85k stars 215 forks source link

Studentbrad/rot user agent #30

Closed studentbrad closed 5 years ago

studentbrad commented 5 years ago

I just made every provider have a different user-agent when using requests. This is because most sites can detect scraping applications. So, if you add a user-agent it simply masks your search with some browser and system variables.