specific proxy not working on cpr but working fine on curl

[bleriaf]

Description

i have a proxy that works on curl however it does not work on cpr, i know the issue and it's due to the fact that the proxy's url starts with http:// while i'm doing a request to an https:// website, however this doesn't seem to be an issue for curl but only for cpr due to this: https://github.com/libcpr/cpr/blob/1986262ba4e0cb052161e9e7919aef5ef08217f0/cpr/session.cpp#L132 that check should be removed and make it use the protocol we specify in the first item of the map in "SetProxies".

and yes i made sure the proxy works on curl (ip address changes, etc)

Example/How to Reproduce

const auto session = std::make_shared<cpr::Session>();
    session->SetProxies( { { "http", "private.com:11111" } } );
    session->SetProxyAuth( { { "http", cpr::EncodedAuthentication{ "user", "pw" } } } );
    session->SetUrl( "https://api.ipify.org/?format=json" );
    session->SetVerifySsl( false );

    const auto r = session->Get( );
    std::cout << r.text << std::endl;

Possible Fix

this code: https://github.com/libcpr/cpr/blob/1986262ba4e0cb052161e9e7919aef5ef08217f0/cpr/session.cpp#L132 should be changed so it listens to the protocol we specify in SetProxies instead of the url but i don't know if thats gonna work good with the system thats in place currently since with the current system one can have a http, a socks4 unique proxies so maybe a better fix is to add another parameter to change in setProxies to specify the protocol (can be optional and if not set, continue to use the automatic protocol solver from the url like currently)?

Where did you get it from?

GitHub (branch e.g. master)

Additional Context/Your Environment

• OS: windows
• Version: last

[bleriaf]

actually i have found the issue and it wasn't what i thought it was, with the issue i had i could have simply changed the protocol from "http" to "https", but i thought that wasn't the issue since i already tried that. so i went and tried to do some changes to cpr and what made the proxy work was to comment this line of code https://github.com/libcpr/cpr/blob/1986262ba4e0cb052161e9e7919aef5ef08217f0/cpr/session.cpp#L136

however after this change using "CURLOPT_PROXYUSERPWD" did not work so i had to to setopt for CURLOPT_PROXYUSERNAME & CURLOPT_PROXYPASSWORD separately for it work

no idea what that does however that helped and now the proxy works

code which works inside cpr (i made so proxyauth returns a pair of both username and password instead of a whole string):

const std::string protocol = url_.str().substr(0, url_.str().find(':'));
    if (proxies_.has(protocol)) {
        curl_easy_setopt(curl_->handle, CURLOPT_PROXY, proxies_[protocol].c_str());
        if (proxyAuth_.has(protocol)) {
            curl_easy_setopt(curl_->handle, CURLOPT_PROXYUSERNAME, proxyAuth_[protocol].first.c_str());
            curl_easy_setopt(curl_->handle, CURLOPT_PROXYPASSWORD, proxyAuth_[protocol].second.c_str());
        }
    }

[COM8]

Thank you very much for taking the time to look into this!

The issue could be the following:

CURLAUTH_ANY
This is a convenience macro that sets all bits and thus makes libcurl pick any it finds suitable. libcurl will automatically select the one it finds most secure. 

Perhaps your proxy server does not support a "safe" authentication method. Could you please try the following:

session->SetVerbose(cpr::Verbose{true});

So we get verbose curl output without commenting curl_easy_setopt(curl_->handle, CURLOPT_PROXYAUTH, CURLAUTH_ANY); out.

[bleriaf]

found another proxy which doesn't work with CURLOPT_PROXYAUTH set to CURLAUTH_ANY, this is the verbose output:

*   Trying 3.**.**.140:****...
* Connected to (nil) (3.**.**.140) port **** (#0)
* allocate connect buffer
* Establish HTTP proxy tunnel to api.ipify.org:443
> CONNECT api.ipify.org:443 HTTP/1.1
Host: api.ipify.org:443
Proxy-Connection: Keep-Alive

< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: Basic realm="LPM"
< Proxy-agent: ****(censured myself)-Proxy
<
* Establish HTTP proxy tunnel to api.ipify.org:443
* Proxy auth using Basic with user '****(censured myself)'
> CONNECT api.ipify.org:443 HTTP/1.1
Host: api.ipify.org:443
Proxy-Authorization: Basic ****(censured myself)
Proxy-Connection: Keep-Alive

* Proxy CONNECT aborted
* CONNECT phase completed
* Closing connection 0

[Sedeniono]

@bleriaf , @COM8 You have removed the curl_easy_setopt(curl_->handle, CURLOPT_PROXYAUTH, CURLAUTH_ANY); call. But doesn't that mean that curl will then use CURLAUTH_BASIC always? If we are on Windows and the proxy requires authentication, the user can enter the credentials in the Windows credential manager. Before this change, cpr would then use CURLAUTH_ANY, which would include CURLAUTH_NTLM, which would cause curl to pick up the credentials from Windows. Since you removed CURLAUTH_ANY, I think this no longer happens? Doesn't the change in this issue here basically cause all proxies requiring authentication to stop working, except the ones that accept CURLAUTH_BASIC?

On the other hand, I don't understand why removing CURLAUTH_ANY could fix the original problem from this issue here. According to its documentation I think CURLAUTH_ANY should work for "basic" proxies, too? Am I interpreting the output you posted correct that with CURLAUTH_ANY the connection to the proxy was established successfully, but for some unknown reason any further communication failed?

[COM8]

The problem is, we don't have a proper way to test this. For me on Linux those changes worked as well as the old way was working. So, if someone on Windows and/or MacOS has a better way of doing this, you are always welcome to create a PR or provide ideas for how to test it easily inside the CI.