After looking into this Crawlee issue, it seems that we handle HTTP requests routed through HTTPS proxy (the request to the proxy server goes over TLS) incorrectly.
The logic picking the correct Agent for the proxy / request combination is here(reader - open this, check it out for 10 seconds and come back, please).
This tells us that different requests are routed through different Agents.
Now, can I get another pair of eyes to confirm what is happening with the different Agents in this file?
From what it seems to me:
HttpRegularProxyAgent: sends a basic (GET / POST etc.) HTTP (as it extends http.Agent) request to the proxy. The target URL is a part of the pathname.
\
This is the reason why the HTTP traffic over HTTPS proxy fails - this Agent is used and tries to send bare HTTP requests to an HTTPS proxy.
HttpProxyAgent: opens an HTTP - or HTTPS, depending on the proxy url - CONNECT tunnel to the proxy server and pushes the request through that.
HttpsProxyAgent: literally calls HttpProxyAgent, but it extends from the https.Agent. I'm not too sure how this is different from the one above.
It seems we have two attributes to care for:
HTTP / HTTPS
target url as a part of the pathname / HTTP CONNECT method
The former we can simply solve by checking the proxy/target URL protocols - the latter seems like a thing dependent on the proxy used. Is there any way to find out whether the proxy server supports only the pathname / the CONNECT method? Is one of them (much) more prevalent? Can HTTPS traffic go only over CONNECT? (this I would assume, otherwise it sounds like some really bad MITM opportunity). But even in that case, what do we pick for the (now broken) HTTP request over HTTPS proxy?
After looking into this Crawlee issue, it seems that we handle HTTP requests routed through HTTPS proxy (the request to the proxy server goes over TLS) incorrectly.
The logic picking the correct Agent for the proxy / request combination is here (reader - open this, check it out for 10 seconds and come back, please).
This tells us that different requests are routed through different Agents.
Now, can I get another pair of eyes to confirm what is happening with the different Agents in this file?
From what it seems to me:
HttpRegularProxyAgent: sends a basic (GET / POST etc.) HTTP (as it extendshttp.Agent) request to the proxy. The target URL is a part of thepathname. \ This is the reason why the HTTP traffic over HTTPS proxy fails - this Agent is used and tries to send bare HTTP requests to an HTTPS proxy.HttpProxyAgent: opens an HTTP - or HTTPS, depending on the proxy url -CONNECTtunnel to the proxy server and pushes the request through that.HttpsProxyAgent: literally callsHttpProxyAgent, but it extends from thehttps.Agent. I'm not too sure how this is different from the one above.It seems we have two attributes to care for:
The former we can simply solve by checking the proxy/target URL protocols - the latter seems like a thing dependent on the proxy used. Is there any way to find out whether the proxy server supports only the
pathname/ theCONNECTmethod? Is one of them (much) more prevalent? Can HTTPS traffic go only over CONNECT? (this I would assume, otherwise it sounds like some really bad MITM opportunity). But even in that case, what do we pick for the (now broken) HTTP request over HTTPS proxy?