Include "Bot" token in default user agent string

typesense / typesense-docsearch-scraper

A fork of Algolia's awesome DocSearch Scraper, customized to index data in Typesense (an open source alternative to Algolia)

https://typesense.org/docs/guide/docsearch.html

Other

95 stars 35 forks source link

Include "Bot" token in default user agent string #48

Closed Krinkle closed 9 months ago

Krinkle commented 9 months ago

Reflect current branding and project name

Include URL for more information.

Include "Bot" token to help various traffic stats and aggregator tools to correctly categorise as spider/bot, even if it doesn't know about typesense-docsearch-scraper specifically. The most common token that these kinds of aggregator and analyzer tools look for is "bot".

PR Checklist

[x] I have read and signed the Contributor License Agreement.

jasonbosco commented 9 months ago

This is now available in 0.9.0 of the scraper.