Using curl under windows as preferred downloader does not work

[fteicht]

Hi,

Being behind a corporate proxy and having many issues with wininet in package control (especially invalid certificate errors), I switched to curl. Under the command prompt I have no problem to fetch the package repository with this command:

curl --proxy PROXYSERVER:PROXYPORT --proxy-user PROXYUSER:PROXYPASSWORD -o channel_v3.json https://packagecontrol.io/channel_v3.json

But it does not work with package control where my settings are:

{ "debug": true, "downloader_precedence": { "windows": ["curl"], "osx": ["urllib"], "linux": ["urllib", "curl", "wget"] }, "http_proxy": "PROXYSERVER:PROXYPORT", "https_proxy": "PROXYSERVER:PROXYPORT", "proxy_username": "PROXYUSER", "proxy_password": "PROXYPASSWORD" }

I get immediately the following debug as if package control did not even try to launch curl:

// DEBUG BEGIN Package Control: Fetching list of available packages Platform: windows-x64 Sublime Text Version: 3083 Package Control Version: 3.1.2 error: Package Control

None of the preferred downloaders can download https://packagecontrol.io/channel_v3.json.

This is usually either because the ssl module is unavailable and/or the command line curl or wget executables could not be found in the PATH.

If you customized the "downloader_precedence" setting, please verify your customization. Package Control: None of the preferred downloaders can download https://packagecontrol.io/channel_v3.json. This is usually either because the ssl module is unavailable and/or the command line curl or wget executables could not be found in the PATH. If you customized the "downloader_precedence" setting, please verify your customization. // DEBUG END

Please note that everything works fine under linux with curl behind the same corporate proxy. Any idea of what is going wrong?

Thanks! Florent

[FichteFoll]

Taking a wild guess here, but Is curl on your PATH?

[wbond]

I would try urllib before curl on Windows.

[fteicht]

I tried urllib but it fails with an SSL certificate error. And yes, curl is on my PATH.

[wbond]

Also, if you are getting invalid certificate errors with a proxy, it may be that your proxy is MITM'ing you. If you provide a full debug log, there may be ways to work around that. For instance, if you know your proxy is inspecting all of your traffic and is generating SSL certificates on the fly, you could add the proxies certificate as a trusted root.

However, if your proxy is just presenting a certificate that is invalid for the domain you are downloading from, Package Control does not currently have a way to work around that, and all of the downloaders will fail. This is by design since it helps ensure you are actually downloading code from where you requested it from.

[wbond]

If curl is in your PATH, then it sounds like you may need to dive into the code and figure out why it doesn't look like it is even trying to use it.

[wbond]

@fteicht Ping re: proxy certificate/debug log

[fteicht]

@wbond How can I trace the logs of my company's proxy? Are you aware of a software which allows me to do that? By the way, I have no trouble downloading https://packagecontrol.io/channel_v3.json under firefox. No security issue is reported.

[wbond]

You would need to turn on the Package Control debug log and switch back to the default downloader configuration. That would give us info about why WinINet is not working.

[fteicht]

OK, thanks wbond for your help. Here is the debug log:

Package Control: Fetching list of available packages Platform: windows-x64 Sublime Text Version: 3083 Package Control Version: 3.1.2 Package Control: Download Debug URL: https://packagecontrol.io/channel_v3.json Timeout: 30 Resolved IP: [Errno 11004] getaddrinfo failed Package Control: Attempting to use Urllib downloader due to WinINet error: Error downloading channel. (errno 12045) during HTTP write phase of downloading https://packagecontrol.io/channel_v3.json. Package Control: Download Debug URL: https://packagecontrol.io/channel_v3.json Timeout: 30 Resolved IP: [Errno 11004] getaddrinfo failed Package Control: Urllib Debug Proxy http_proxy: XXX:XXX https_proxy: XXX:XXX proxy_username: XXX proxy_password: XXX Package Control: Generating new CA bundle from system certificate store Package Control: Exported certificate "Microsoft Root Certificate Authority" Package Control: Skipping certificate "Thawte Timestamping CA" since it uses the signature algorithm md5WithRSAEncryption Package Control: Skipping certificate "Microsoft Root Authority" since it uses the signature algorithm md5WithRSAEncryption Package Control: Exported certificate "Microsoft Root Certificate Authority 2011" Package Control: Skipping certificate "Microsoft Authenticode(tm) Root Authority" since it is no longer valid Package Control: Exported certificate "Microsoft Root Certificate Authority 2010" Package Control: Skipping certificate "Copyright (c) 1997 Microsoft Corp." since it is no longer valid Package Control: Skipping certificate "NO LIABILITY ACCEPTED, (c)97 VeriSign, Inc." since it is no longer valid Package Control: Exported certificate "UTN-USERFirst-Object" Package Control: Exported certificate "Equifax Secure Certificate Authority" Package Control: Skipping certificate "GTE CyberTrust Global Root" since it uses the signature algorithm md5WithRSAEncryption Package Control: Skipping certificate "Class 3 Public Primary Certification Authority" since it uses the signature algorithm md2WithRSAEncryption Package Control: Skipping certificate "Class 3 Public Primary Certification Authority" since it is no longer valid Package Control: Exported certificate "VeriSign Class 3 Public Primary Certification Authority - G5" Package Control: Exported certificate "Network Solutions Certificate Authority" Package Control: Skipping certificate "airforge.services.its.corp" since it is no longer valid Package Control: Exported certificate "*.mykds.com" Package Control: Skipping certificate "CORP-IPSEC-CA" since it is no longer valid Package Control: Exported certificate "XXX Sub CA" Package Control: Exported certificate "XXX HQ Sub CA" Package Control: Exported certificate "XXX" Package Control: Exported certificate "XXX" Package Control: Exported certificate "XXX Root CA 2" Package Control: Exported certificate "XXX" Package Control: Exported certificate "UTN-USERFirst-Hardware" Package Control: Exported certificate "XXX S2 Root CA 1" Package Control: Exported certificate "XXX" Package Control: Exported certificate "XXX Sub CA" Package Control: Exported certificate "XXX Root CA" Package Control: Exported certificate "AddTrust External CA Root" Package Control: Skipping certificate "XXX" since it is no longer valid Package Control: Skipping certificate "XXX" since it is no longer valid Package Control: Exported certificate "HQ-CA" Package Control: Skipping certificate "test" since it is no longer valid Package Control: Exported certificate "VeriSign Class 3 Code Signing 2010 CA" Package Control: Skipping certificate "Root Agency" since it uses the signature algorithm md5WithRSAEncryption Package Control: Exported certificate "www.verisign.com/CPS Incorp.by Ref. LIABILITY LTD.(c)97 VeriSign" Package Control: Skipping certificate "Microsoft Windows Hardware Compatibility" since it is no longer valid Package Control: Exported certificate "Entrust Certification Authority - L1C" Package Control: Exported certificate "XXX pscCA1" Package Control: Exported certificate "XXX ecoCA1" Package Control: Skipping certificate "XXX" since it is no longer valid Package Control: Skipping certificate "XXX" since it is no longer valid Package Control: Exported certificate "HQ-CA" Package Control: Skipping certificate "test" since it is no longer valid Package Control: Finished generating new CA bundle at XXX\AppData\Roaming\Sublime Text 3\Packages\User\Package Control.system-ca-bundle Package Control: Regenerated the merged CA bundle from the system and user CA bundles Package Control: Urllib HTTPS Debug General Connecting to XXX on port XXX Package Control: Urllib HTTPS Debug Write CONNECT packagecontrol.io:443 HTTP/1.1 Proxy-Connection: Keep-Alive User-Agent: Package Control v3.1.2 Host: packagecontrol.io:443 Package Control: Urllib HTTPS Debug Read HTTP/1.1 407 Proxy Authentication Required Proxy-Authenticate: NEGOTIATE Proxy-Authenticate: NTLM Proxy-Authenticate: BASIC realm="Auth_AD_IW" Cache-Control: no-cache Pragma: no-cache Content-Type: text/html; charset=utf-8 Proxy-Connection: close Set-Cookie: BCSI-CS-550c0ee8fe694647=2; Path=/ Connection: close Content-Length: 849 Package Control: Urllib HTTPS Debug Write CONNECT packagecontrol.io:443 HTTP/1.1 Proxy-Connection: Keep-Alive User-Agent: Package Control v3.1.2 Proxy-Authorization: Basic NDU1MjI0NTc6JDNwTU1hMiU= Host: packagecontrol.io:443 Package Control: Urllib HTTPS Debug Read HTTP/1.1 200 Connection established Package Control: Urllib HTTPS Debug General Upgrading connection to SSL using CA certs file at XXX\AppData\Roaming\Sublime Text 3\Packages\User\Package Control.merged-ca-bundle Package Control: Error downloading channel. HTTP exception InvalidCertificateException (Host XXX:XXX returned an invalid certificate ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:550))) downloading https://packagecontrol.io/channel_v3.json.

[wbond]

So now that you know the issue is related to the proxy and certificates you need to figure out if there is a CA cert you need for your proxy, or some other way to get such connections working.

You could use the Firefox certificate pane to see what chain Firefox is using to validate packagecontrol.io. That may help in identifying if you proxy is changing things.

[wikt0r]

Quote from here: https://github.com/wbond/package_control/issues/1334#issuecomment-375198525

On Windows, the curl or wget tips not work. Because, PackageControl looks for curl instead of curl.exe.

After I read this I simply copied curl.exe to curl and everything works fine.