issues
search
monperrus
/
crawler-user-agents
Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome :star:
MIT License
1.19k
stars
254
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Added user agents from Apify / crawlee-python fingerprint dataset
#377
brentclark
opened
2 weeks ago
1
Add KeybaseBot
#376
starius
closed
2 weeks ago
0
Update crawler-user-agents.json with AcademicBotRTU
#375
omitech
closed
2 weeks ago
2
Disambiguate http clients from crawlers/bots
#374
srstsavage
opened
1 month ago
2
Add facebookcatalog, meta-externalagent, and meta-externalfetcher
#373
srstsavage
closed
1 month ago
1
Add OpenAI Search Bot
#372
elisa-luo
closed
1 month ago
1
Consider adding `.mjs` export
#370
DamianGlowala
opened
1 month ago
1
feat: Add `id` field to each item
#369
blaine-arcjet
closed
2 months ago
2
Improve Python usage harness
#368
jribbens
closed
2 months ago
4
chore: Update some links & instances
#367
blaine-arcjet
closed
2 months ago
1
Request for new instances
#366
petrospap
opened
2 months ago
1
chore: Normalize formatting of the JSON file
#365
blaine-arcjet
closed
2 months ago
4
chore: Combine seekport rules
#364
blaine-arcjet
closed
2 months ago
0
feat: Add Vercelbot and vercel-screenshot user-agents
#363
blaine-arcjet
closed
2 months ago
1
Add Chrome-Lighthouse instance
#362
acrollet
closed
2 months ago
1
Misidentified user-agents as bots
#361
sagrawal31
opened
3 months ago
2
Few Google bots are not considered
#360
sagrawal31
opened
3 months ago
1
patterns json: add support for GroupMe bot
#359
bentsi
closed
5 months ago
2
Adding Monsidobot and removing whitespace
#358
theodorec2
closed
5 months ago
1
Add and use `requirements.txt`
#357
ericcornelissen
closed
6 months ago
1
Fix pattern of Nutch crawler
#356
haumacher
closed
6 months ago
1
Add ChatGPT-User - another bot from openai.com
#355
haumacher
closed
6 months ago
0
Add ClaudeBot
#354
haumacher
closed
6 months ago
0
golang: speedup using TRIE
#353
starius
closed
2 weeks ago
10
Bump tough-cookie and pacote
#352
dependabot[bot]
closed
7 months ago
0
Add npm dependencies with lockfile
#351
ericcornelissen
closed
7 months ago
4
Suggestion: test against false positives
#350
starius
closed
7 months ago
2
add date validation
#349
monperrus
closed
7 months ago
0
Add Golang package
#348
starius
closed
7 months ago
10
Use Node.js v20 in CI to have npm CLI version with support for --provenance
#347
ericcornelissen
closed
7 months ago
0
Update BuiltWith
#346
VoidMonk
closed
7 months ago
1
Add Audisto Crawler and PerplexityBot
#345
VoidMonk
closed
7 months ago
1
Add GeedoBot
#344
abushyk
closed
9 months ago
1
Add BuiltWith bot
#343
flopana
closed
9 months ago
1
port CI to GithubActions
#342
monperrus
closed
7 months ago
4
Add Expanse and InternetMeasurement crawler
#341
flopana
closed
9 months ago
1
Add PhxBot and ImagesiftBot
#340
dspinellis
closed
10 months ago
1
Add AI agents
#339
httpjamesm
opened
10 months ago
1
Escape dots in patterns
#338
giuscris
closed
7 months ago
5
Fix for SeznamBot detection
#337
synhershko
closed
11 months ago
1
Add DataForSeoBot
#336
abushyk
closed
1 year ago
1
added WordPress crawler
#335
petrokrupenia
closed
1 year ago
8
Add WordPress Bot/Crawler
#334
petrokrupenia
closed
10 months ago
2
Merge all yandex bots together
#333
fekir
closed
1 year ago
1
Add multiple bots found in the logs
#332
fekir
closed
1 year ago
7
Add Nicecrawler
#331
fekir
closed
1 year ago
1
Add Marginalia and YouBot
#330
dspinellis
closed
1 year ago
1
I noticed a recent bot which I believe is coming from Tencent but can't find documentation for - Ottabot
#329
daviddfm
opened
1 year ago
1
Add StractBot
#328
dspinellis
closed
1 year ago
1
Add patent crawler
#327
dspinellis
closed
1 year ago
0
Next