There are a lot of errors in the log

dtrungtin / actor-airbnb-scraper

Airbnb Scraper actor is designed to extract most of publicly available data for home listings

https://apify.com/dtrungtin/airbnb-scraper

27 stars 22 forks source link

There are a lot of errors in the log #25

Closed zpelechova closed 3 years ago

zpelechova commented 3 years ago

It might put off users even when the scrape is succesfull in the end. https://my.apify.com/view/runs/SO2pxPNn0CqBQzbsb

dtrungtin commented 3 years ago

I don't think it's bug. These errors are the requests to Airbnb server returning the error. It may bot protection, network issues, etc. However, we have some retries on such request. Solution:

Hide the errors (I don't like this option)
Choose other better proxies as Residential to run the actor

zpelechova commented 3 years ago

YOu may hide some errors or make it part of debug log - maybe a best option? As this way the user thinks it is not wirking especially with airbnb where first results takes 30 minutes to get.

dtrungtin commented 3 years ago

ERROR BasicCrawler: handleRequestFunction failed, reclaiming failed request back to the list or queue

It's error log, not a debug log. Like other actors, If there is an error when crawling 1 request URL, it should be logged as ERROR and then retry to crawling this request URL again.