search

InteractiveAdvertisingBureau / adstxtcrawler

A reference implementation in python of a simple crawler for Ads.txt
180 stars 115 forks source link

User agent #12

Closed nomoregrapes closed 3 years ago

nomoregrapes commented 6 years ago

What user agent will this crawler report as?

Please can there be a setting to specify information as to who is running the crawler, and put this in the useragent. This way, website owners can contact those responsible if they are being excessively crawled.

mrwilson commented 6 years ago

It looks from the code like the user-agent is being set as "AdsTxtCrawler/1.0; +https://github.com/InteractiveAdvertisingBureau/adstxtcrawler"

(https://github.com/InteractiveAdvertisingBureau/adstxtcrawler/blob/master/adstxt_crawler.py#L119)

But there's currently no feature to support customising the user-agent.

nealrichter commented 3 years ago

This is trivial to change in the code itself. One would expect any scaled ads company to do it...