search

flathunters / flathunter

A bot to help people with their rental real-estate search. 🏠🤖
GNU Affero General Public License v3.0
841 stars 180 forks source link

Current state of solving `IS24 bot detection has identified our script as a bot - we've been blocked` #589

Closed robertszooba closed 1 month ago

robertszooba commented 4 months ago

This is not the first issue regarding this error. However, browsing through various threads, I am unable to abstract a specific step-by-step guide to solve this problem. Therefore, I would like to ask what needs to be done to circumvent the bot detection. The relevant parts of my config.yaml are:

loop:
  active: yes
  sleeping_time: 600

captcha:
  imagetyperz:
    token: ABC
  driver_arguments:
    - "--headless"

immoscout_cookie: abc
codders commented 4 months ago

Hi @robertszooba,

Thanks for the report. I guess the reason you don't see a step by step guide is that we don't really know exactly how to bypass the detection.

It is certainly necessary to have captcha support setup. For the driver_arguments, the "default" set is:

      "--no-sandbox",
      "--headless",
      "--disable-gpu",
      "--remote-debugging-port=9222",
      "--disable-dev-shm-usage",
      "--window-size=1024,768"

But different people have different results on different platforms. How the IS24 blocking specifically works is kept secret for obvious reasons, and the flathunter software does its best to emulate a real user-driven browser.

Besides that I don't really know what to recommend. If you have been making a lot of requests, it might just be that your IP address has been flagged as bad - are you able to browse the site manually in a private window?

If manual browsing works, you can try and compare the requests that your browser makes with the the requests the flathunter makes. If you remove the --headless argument, you will be able to interact with the automatic browser, and that might give you some clues.

Hope that helps - maybe other users also have tips.

Arthur

robertszooba @.***> schrieb am Sa., 11. Mai 2024, 11:53:

This is not the first issue regarding this error. However, browsing through various threads, I am unable to abstract a specific step-by-step guide to solve this problem. Therefore, I would like to ask what needs to be done to circumvent the bot detection. The relevant parts of my config.yaml are:

loop: active: yes sleeping_time: 600

captcha: imagetyperz: token: ABC driver_arguments:

  • "--headless"

immoscout_cookie: abc

— Reply to this email directly, view it on GitHub https://github.com/flathunters/flathunter/issues/589, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAEK5XRQBFV3OYOB2K5AVDZBXS77AVCNFSM6AAAAABHRZNYX6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGI4TAOBRGMZTGMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>