search

orangecoding / fredy

:heart: Fredy - [F]ind [R]eal [E]states [D]amn Eas[y] - Fredy will constantly search for new listings on sites like Immoscout or Immowelt and send new results to you, so that you can focus on more important things in life ;)
http://www.orange-coding.net
MIT License
209 stars 54 forks source link

Immoscout with ScrapingAnt: Try a data-center request first before using residential proxies #54

Closed tomburgert closed 2 years ago

tomburgert commented 2 years ago

Hi,

first of all, thank you for your amazing work. I have a small feature request: would it be possible when scraping immoscout with the help of ScrapingAnt to try to use a data-center proxy call before using residential proxies? The current call consumes 250 credits, which means even with paying a monthly plan of 19$, only 400 calls would be allowed per month. Considering the very short reaction times that are needed to find a flat (for popular cities emails should be sent within 5 minutes) roughly 13 calls per day are just not enough. By using data-center proxies there would be chance of making 25x more calls. Otherwise, with the current consumption of credits, using fredy for immoscout crawling would not help a lot, unless they are able to spend 250$ a month for the service of ScrapingAnt.

Best, Tom

orangecoding commented 2 years ago

Yep I'm planning to actually rework some parts around the immoscout scraping. I'll keep you posted

orangecoding commented 2 years ago

I talked to the founder of scraping ant. They have improved the detection rate on their end and I added this retry. https://github.com/orangecoding/fredy/commit/676d48807a71994b8350d45a1a4779f1fa27dbf3

orangecoding commented 2 years ago

I will now first try it with a "regular" proxy, then switch to residential if detected. Keep in mind, this is the case in like 99 % of all cases.

tomburgert commented 2 years ago

Thank you so much for your quick support!