Closed rentalhost closed 1 year ago
Thanks for this. Go ahead and create a PR for the agents i have ticked. Will look into the other at a later date 👍
Please also add "LieBaoFast" Chinese scrapers.
Take a look at this: https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/
@mtshare would you like to submit a PR to add that bot?
Is the corresponding pr merged?
Is the corresponding pr merged?
No PR was ever submitted 😔
Sorry, I ended up waiting for the analysis of the other User Agents before sending the PR. Anyway, if someone can send me a PR, I appreciate it (I'm a little bit in trouble now).
If no one is assigned, can I push a pr?
Should the take into consideration all the user agents assigned over here?
Just the ones that have been ticked 👍🏻
I have found some new possibilities, but I not right what should be added or not, or even if it is really a crawler. So I am creating this issue with checkboxes to you decide. I will make a PR after the decision.
Dangerous possibility (eg. cracker):
The quoted user-agents are as was received by my logger.
High probability:
Microsoft Office happen when you copy some content from web to it, then in some cases it will download content from original page. Same for Mashup.
Medium probability:
Can't found what it mean, but not seems to be a real browser.
Low probability:
I don't know what mean CFNetwork, but is related to Apple. Outlook and OC is very similar to Microsoft Office case.