User-Agent not set - Githubissues

edoardottt / cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

https://edoardoottavianelli.it

GNU General Public License v3.0

1.49k stars 152 forks source link

Closed edoardottt closed 5 months ago

edoardottt commented 5 months ago

seems like the user agent is not set properly.

Should be investigated.

edoardottt commented 5 months ago

The User Agent is set for regular requests during crawling. However, colly performs some CONNECT requests with the default Go User Agent.

Example:

CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com:443
Connection: close
User-Agent: Go-http-client/1.1