JayBizzle / Crawler-Detect

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
https://crawlerdetect.io
MIT License
1.94k stars 254 forks source link

Remove 'okhttp' user agent as it is being used by home-assistant android app #437

Closed e-minguez closed 3 years ago

e-minguez commented 3 years ago

https://github.com/JayBizzle/Crawler-Detect/blob/master/raw/Crawlers.txt#L800

bunkerized-nginx uses these lists, and it blocks access to home-assistant when trying to log in as:

2021/02/16 12:55:07 [warn] 24951#24951: *35 [lua] access_by_lua(main-lua.conf:166):80: [BLOCK] User-Agent okhttp/4.9.0 is blacklisted, client: 10.0.2.100, server: www.example.com, request: "POST /auth/token HTTP/2.0", host: "www.example.com"

Additional information

https://github.com/home-assistant/android/search?q=okhttp&type=code

JayBizzle commented 3 years ago

Thanks for the issue report.

Im 50/50 on what to do here. I understand this is causing an issue in a 3rd party app, but for the intended use here, it certainly seems like it should be classed as a bot visit.

😕

e-minguez commented 3 years ago

Thanks for the issue report.

Im 50/50 on what to do here. I understand this is causing an issue in a 3rd party app, but for the intended use here, it certainly seems like it should be classed as a bot visit.

confused

I'm trying to convince the home-assistant folks to customize the android app requests to avoid using okhttp and use a custom one instead https://github.com/home-assistant/android/issues/1364 :)

JayBizzle commented 3 years ago

Okay, thanks, i'll keep an eye on it 👍

dshokouhi commented 3 years ago

@JayBizzle we have now customized our user agent string to avoid using the default one provided by okhttp

JayBizzle commented 3 years ago

Good stuff. Thanks for letting us know 👍