User-Agent not set - Githubissues

edoardottt / cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

https://edoardottt.com/

GNU General Public License v3.0

1.54k stars 158 forks source link

Closed edoardottt closed 8 months ago

edoardottt commented 8 months ago

seems like the user agent is not set properly.

Should be investigated.

edoardottt commented 8 months ago

The User Agent is set for regular requests during crawling. However, colly performs some CONNECT requests with the default Go User Agent.

Example:

CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com:443
Connection: close
User-Agent: Go-http-client/1.1