search

edoardottt / cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more
https://edoardoottavianelli.it
GNU General Public License v3.0
1.49k stars 152 forks source link

Proxy settings not honored #122

Closed ocervell closed 5 months ago

ocervell commented 1 year ago

It seems that proxies are not honored, by looking at Wireshark traffic I see some requests not going through any proxy.

I think this is related to https://github.com/gocolly/colly/issues/392

We probably need to set 

c.WithTransport(&http.Transport{
  DisableKeepAlives: true,
})

in the code here

edoardottt commented 1 year ago

Hi @ocervell, thanks for the issue! Have you tried this solution? Because in the issue you linked it seems it worked just for one person. Moreover, disabling keep-alive connections will affect performance, so we shoud be sure it works fine

ocervell commented 1 year ago

I am sure this is an issue, but i'm not sure yet about the solution. Indeed disable keep-alive might decrease performance, so we should not do it in non-proxy modes. imho when you pass proxy, you want to be sure that no traffic leaks outside the proxy.

edoardottt commented 1 year ago

Absolutely agree. When I'll have enough time to make some tests I'll take a deep look about that :)

As always, thanks for your help, really appreciated ❤️

edoardottt commented 5 months ago

@ocervell tentative for the fix here > https://github.com/edoardottt/cariddi/issues/143#issuecomment-2016560137

edoardottt commented 5 months ago

Partially fixed in version 1.3.3.

Many targets work fine with new proxy settings, however there could be problems in certain types of target. e.g.:

Before this fix there was a clear problem with proxies, no target was working. After this fix a lot of targets can be crawled using a proxy.

If anyone has a better solution I'm all ears, just open an Issue / Pull Request