search

urbanadventurer / WhatWeb

Next generation web scanner
https://www.morningstarsecurity.com/research/whatweb
GNU General Public License v2.0
5.58k stars 909 forks source link

using proxy + HTTPS ends with SSL session is not started yet #389

Open xambroz opened 1 year ago

xambroz commented 1 year ago

Hello, there seems to be some issue with (may be race condition of slow non-caching) http proxy when requesting HTTPS sites.

Without proxy

$ whatweb https://www.root.cz/ 
https://www.root.cz/ [200 OK] Cookies[iinfo-sid], Country[CZECH REPUBLIC][CZ], Email[b945dfd40ee14a049221190a7f57fe2d@sentry.iinfo.cz,erik.dvorak@iinfo.cz], Frame, HTML5, HTTPServer[nginx/1.14.2], HttpOnly[iinfo-sid], IP[91.213.160.188], JQuery[2.1.4], Open-Graph-Protocol[article][160071904046461], PasswordField[us_pwd], Script[text/javascript], Strict-Transport-Security[max-age=63072000; includeSubDomains; preload], Title[Root.cz - informace nejen ze světa Linuxu], UncommonHeaders[x-networkmanager-status,content-security-policy,x-backend-server,referrer-policy,access-control-allow-header], X-Backend[web2 D=126273 t=1677330589429008, web3-new], nginx[1.14.2]

With proxy

$ whatweb --proxy localhost:8118 https://www.root.cz/ 
ERROR Opening: https://www.root.cz/ - SSL session is not started yet

Curl / wget ang others go well through the proxy (privoxy in my case)

$ curl -I --proxy localhost:8118 https://www.root.cz/ 2>&1|head -n 30
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 200 Connection established

HTTP/2 200 
server: nginx/1.14.2
date: Sat, 25 Feb 2023 13:13:42 GMT
content-type: text/html; charset=UTF-8
strict-transport-security: max-age=63072000; includeSubDomains; preload
x-networkmanager-status: online
set-cookie: iinfo-sid=gjqti1ulr6a58f2u6mtoknh5cn; expires=Tue, 07-Mar-2023 13:13:42 GMT; Max-Age=864000; path=/; domain=.root.cz; secure; HttpOnly
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
vary: X-Requested-With
x-backend-server: web8 D=422603 t=1677330822338638
referrer-policy: no-referrer-when-downgrade
access-control-allow-header: *
x-backend-server: web3-new

Workaround - It works using proxychains:

$ proxychains whatweb https://www.root.cz/ 
[proxychains] config file found: /home/mambroz/.proxychains/proxychains.conf
[proxychains] preloading /usr/lib64/proxychains-ng/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.16
[proxychains] DLL init: proxychains-ng 4.16
[proxychains] Strict chain  ...  127.0.0.1:8118  ...  91.213.160.188:443  ...  OK
https://www.root.cz/ [200 OK] Cookies[iinfo-sid], Country[CZECH REPUBLIC][CZ], Email[b945dfd40ee14a049221190a7f57fe2d@sentry.iinfo.cz,erik.dvorak@iinfo.cz], Frame, HTML5, HTTPServer[nginx/1.14.2], HttpOnly[iinfo-sid], IP[91.213.160.188], JQuery[2.1.4], Open-Graph-Protocol[article][160071904046461], PasswordField[us_pwd], Script[text/javascript], Strict-Transport-Security[max-age=63072000; includeSubDomains; preload], Title[Root.cz - informace nejen ze světa Linuxu], UncommonHeaders[x-networkmanager-status,content-security-policy,x-backend-server,referrer-policy,access-control-allow-header], X-Backend[web7 D=288462 t=1677330894285181, web3-new], nginx[1.14.2]
[mambroz@czcholn5025023:~] 2023-02-25 14:15:
rloretan commented 10 months ago

Experienced the same issue today.