search

vertexproject / synapse

Synapse Central Intelligence System
Apache License 2.0
353 stars 74 forks source link

[BUG] lib.inet.http.get connects via https instead of specified url protocol when a proxy is set. #3708

Closed jwahsnakupaku closed 5 months ago

jwahsnakupaku commented 5 months ago

Describe the bug When I specify a url to lib.inet.http.get, with a http proxy, it tries to connect via https instead of http.

To Reproduce Specify http url with http proxy;

$lib.inet.http.get('http://netinfo.local/lookup?ip=1.1.1.1', proxy="http://webproxy.local:3128")
#SQUID Logs
1714709924.192      0 <Cortex IP> TCP_DENIED/403 3934 CONNECT netinfo.local:80 - HIER_NONE/- text/html

If I use Curl it works as expected

http_proxy=http://webproxy.local:3128 curl http://webproxy.local/lookup?ip=1.1.1.1
# SQUID Logs
1714709989.807      2 <CURL IP> TCP_MISS/200 681 GET http://netinfo.local/lookup? - HIER_DIRECT/<Netinfo IP> application/json

If I remove the proxy it'll work as expected, eg; $lib.inet.http.get('http://netinfo.local/lookup?ip=1.1.1.1'")

If I use a https url it works;

$lib.inet.http.get('https://netinfo.local/lookup?ip=1.1.1.1', proxy="http://webproxy.local:3128")
# Squid Logs
1714710571.697     21 <Cortex IP>2 TCP_TUNNEL/200 3073 CONNECT netinfo.local:443 - HIER_DIRECT/<Netinfo IP> -

Expected behavior $lib.inet.http.get tries HTTP instead of HTTPS..

Environment (please complete the following information):

Additional context Add any other context about the problem here.

vEpiphyte commented 5 months ago

@jwahsnakupaku Are there any exception logs on the Cortex when you attempt the first connection?

Can you share how your proxy is setup & configured?

jwahsnakupaku commented 5 months ago

Yeah, errors with 403 forbidden.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/python_socks/async_/asyncio/v2/_proxy.py", line 118, in _connect
    await connector.connect(
  File "/usr/local/lib/python3.11/dist-packages/python_socks/_connectors/http_async.py", line 36, in connect
    reply: http.ConnectReply = conn.receive(data)
                               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/python_socks/_protocols/http.py", line 147, in receive
    return ConnectReply.loads(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/python_socks/_protocols/http.py", line 136, in loads
    raise ReplyError(msg, error_code=status_code)
python_socks._protocols.errors.ReplyError: 403 Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/synapse/lib/stormhttp.py", line 465, in _httpRequest
    async with sess.request(meth, url, headers=headers, **kwargs) as resp:
  File "/usr/local/lib/python3.11/dist-packages/aiohttp/client.py", line 1194, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/aiohttp/client.py", line 578, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/aiohttp/connector.py", line 544, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/aiohttp/connector.py", line 911, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/aiohttp/connector.py", line 1204, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/aiohttp_socks/connector.py", line 85, in _wrap_create_connection
    stream = await proxy.connect(
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/python_socks/async_/asyncio/v2/_proxy.py", line 70, in connect
    return await self._connect(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/python_socks/async_/asyncio/v2/_proxy.py", line 131, in _connect
    raise ProxyError(e, error_code=e.error_code)
python_socks._errors.ProxyError: 403 Forbidden

Squid is setup to only allow specific ports to connect, eg; http_access deny CONNECT !SSL_ports If I add 80 to SSL_ports it'll work.

Guess I didn't expect http traffic to go via connect, if that's expected can probably close this.

vEpiphyte commented 5 months ago

The HTTP proxy implementation ( using aiotthp-socks & python-socks ) doesn't distinguish between HTTP / HTTPS - all traffic is routed through the configured proxy using a CONNECT statement. Can you try adding the expected HTTPS ports in your squid configuration?

jwahsnakupaku commented 5 months ago

Yes have confirmed that works, thanks for you help.