Hoxmot
commented
1 year ago I got blocked in the implementation process.
Unfortunately, searching through the apartments has bot protection. The bot protection prevents me from querying the website. I was only expecting bot protection in the application process.
I tried multiple approaches:
- Simple request with Axios to the searching link. I got immediately marked as a bot.
- Using Puppeteer to run a browser and navigate to the website. This landed me on a captcha. I probably won't be able to circumvent the captcha.
- I copied my browser's exact request with all the cookies and headers. The method worked for a brief period, but after some time, the requests also redirected me to a captcha.
The behaviour is as follows: after I navigate to the search link, I land on a bot protection site. It takes a while to validate the browser and the results are shown if the browser passes the protection. Then, when I navigate further, there's no bot protection screen. If I don't browse the website for a while, I get the bot protection again.
I think there might be a cookie valid for a time, marking me a non-bot user. Considering it works when I browse the page, it might get refreshed every now and then or with each request.
I have one more idea, but I don't think it's worth the effort. I can try to find which cookie is important for the bot protection mechanism. Then, I can try to see how the cookie gets refreshed. Once I know that, I can implement a mechanism which updates the cookie with each navigation. Then, the operator of the program would have to initialize the session by adding a valid cookie to the configuration. Then, the program can automatically keep alive the cookie by doing a navigation every X minutes (depending on the TTL of the cookie).
Check how the search is done with immoscout and mimic the same process.
Ensure the API I prepare inside my package is universal enough to add support for other websites later.
I can't use the API immoscout has since it requires contact with the support, so it won't be feasible for the users of the package to configure it themselves.