Zillow crawler stopped working

[thekingofkings]

Zillow crawler cannot crawl data anymore

Issue

Now the crawler get a page asking for recaptcha, because it recognize the python requests not from a browser.

Solution

Mimic browser request by adding User-Agent in the request header?

[thekingofkings]

The zillow also tracts the request frequency and block any IP making dense requests (dim as robot).

Solution: add a time.sleep() to lower the request time.