search

amerkurev / scrapper

Web scraper with a simple REST API living in Docker and using a Headless browser and Readability.js for parsing.
https://scrapper.dev
Apache License 2.0
173 stars 29 forks source link

How can I set name of Crawler? #15

Closed changingmission closed 5 months ago

changingmission commented 5 months ago

is there any way I can set my crawler name/desc etc.?

amerkurev commented 5 months ago

@changingmission sorry, I don't understand what you mean by "set name of crawler." Can you explain it to me in more detail?

changingmission commented 5 months ago

Like google crawler use name : Googlebot

changingmission commented 5 months ago

@amerkurev is there any way? can I set crawler name?

amerkurev commented 5 months ago

@changingmission you can add header for this using the "extra-http-headers" parameter as described in the documentation. Example: extra-http-headers: User-Agent:Googlebot-Image/1.0.

https://github.com/amerkurev/scrapper?tab=readme-ov-file#browser-settings