Closed kaizen89 closed 2 years ago
Hi,
Have you tried it many times? Timeout is often matter of luck. In very bad networks it's possible that 10 sec is never enough to establish the connection (often due to DNS issues).
Increasing the timeout would require to set the curl option CONNECTTIMEOUT
. Unfortunately the curl
package provides API to set it only on the curl_handle
objects. And the readr
package has absolutely no API to set any curl options. And the curl
package doesn't support the R options
API. It makes our job quite difficult when we try to address this in OmnipathR
. I'm still thinking about a solution, but let me know if repeated attempts work.
Best,
Denes
I added an option to set curl connection timeout, you can try it by updating OmnipathR
from the latest commit in its master branch. The default value of this option (omnipath.connect_timeout
) is the same as the default of curl (10 seconds). To raise it to 60 seconds:
library(OmnipathR)
options(omnipath.connect_timeout = 60)
I can't really test if it works as I don't know a server which consistently gives a connect timeout, and the curl
R package lacks the API to inspect handlers.
Thanks for your reactivity, unfortunately after many attempts, setting omnipath.connect_timeout
still same error message.
Hi,
To confirm that indeed the long connection time is responsible for the failure, you could run curl
from the command line (bash) with its default and an increased timeout:
progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
curl -vvv -o 'progeny.tsv' -L $progeny_url
curl --connection-timeout 60 -vvv -o 'progeny-to60.tsv' -L $progeny_url
We expect the first to fail, and the second maybe to succeed. If the second one fails too due to a connect timeout, it means in my opinion, that your connection just stalls and even an infinite timeout couldn't fix the problem. It can happen if let's say the DNS never replies, gives a wrong response, or a firewall swallows all the response, and under many further weird network conditions.
Also my quick hack with the timeout option in OmnipathR
broke its behaviour under certain conditions, hence I suggest to update it again to version 3.5.2
where I fixed these issues.
Both failed unfortunately
$ curl --connect-timeout 60 -vvv -o 'progeny-to60.tsv' -L $progeny_url
* Trying 192.168.160.60...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to (nil) (192.168.160.60) port 3128 (#0)
* Establish HTTP proxy tunnel to omnipathdb.org:443
> CONNECT omnipathdb.org:443 HTTP/1.1
> Host: omnipathdb.org:443
> User-Agent: curl/7.52.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Date: Wed, 25 May 2022 08:44:07 GMT
< Proxy-Connection: Keep-Alive
< Via: 1.1 stentor.curie.net
<
* Proxy replied OK to CONNECT request
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /home/salmon/anaconda2/ssl/cacert.pem
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
0 0 0 0 0 0 0 0 --:--:-- 0:00:59 --:--:-- 0* Operation timed out after 60614 milliseconds with 0 out of 0 bytes received
* Curl_http_done: called premature == 1
0 0 0 0 0 0 0 0 --:--:-- 0:01:00 --:--:-- 0
* Closing connection 0
curl: (28) Operation timed out after 60614 milliseconds with 0 out of 0 bytes received
But it looks like communication has just started before it timed out? This is very extreme, but maybe you could try with even longer connect timeout:
progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
curl --connection-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
curl --connect-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
* Trying 192.168.160.61...
* TCP_NODELAY set
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to (nil) (192.168.160.61) port 3128 (#0)
* Establish HTTP proxy tunnel to omnipathdb.org:443
> CONNECT omnipathdb.org:443 HTTP/1.1
> Host: omnipathdb.org:443
> User-Agent: curl/7.52.1
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Date: Wed, 25 May 2022 10:40:50 GMT
< Proxy-Connection: Keep-Alive
< Via: 1.1 stentor.curie.net
<
* Proxy replied OK to CONNECT request
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: /home/salmon/anaconda2/ssl/cacert.pem
CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
0 0 0 0 0 0 0 0 --:--:-- 0:01:02 --:--:-- 0* Unknown SSL protocol error in connection to omnipathdb.org:443
* Curl_http_done: called premature == 1
0 0 0 0 0 0 0 0 --:--:-- 0:01:03 --:--:-- 0
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to omnipathdb.org:443
This is now a different error. The reasons behind are not clear, but likely a missing certificate on your computer or the incompatibility between the cipher suites supported by your computer and our server. These happen most often due to old client side software. Your curl
is 6 years old, I would recommend to update it. Before, it would be interesting to check the supported protocols and ciphers:
curl --version
curl https://www.howsmyssl.com/a/check
curl -o 'curl-7.52.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'
Then you could check again everything: the supported standards, accessing OmniPath server, and would be nice to test some other server too. Btw, have you ever experienced similar issue with any other webpage, especially if accessed programmatically?
curl --version
curl https://www.howsmyssl.com/a/check
curl -o 'curl-7.83.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'
progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
curl --connection-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
Before update
$ curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy
$ curl https://www.howsmyssl.com/a/check
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option
$curl -o 'curl-7.52.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 264 100 264 0 0 142 0 0:00:01 0:00:01 --:--:-- 142
100 269 100 269 0 0 70 0 0:00:03 0:00:03 --:--:-- 197
100 35438 0 35438 0 0 5814 0 --:--:-- 0:00:06 --:--:-- 22760
After update
$ curl --version
curl 7.70.0 (x86_64-pc-linux-gnu) libcurl/7.70.0 OpenSSL/1.1.1 zlib/1.2.11
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets
$ curl https://www.howsmyssl.com/a/check
{"given_cipher_suites":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_EMPTY_RENEGOTIATION_INFO_SCSV"],"ephemeral_keys_supported":true,"session_ticket_supported":false,"tls_compression_supported":false,"unknown_cipher_suite_supported":false,"beast_vuln":false,"able_to_detect_n_minus_one_splitting":false,"insecure_cipher_suites":{},"tls_version":"TLS 1.3","rating":"Probably Okay"}
$ curl -o 'curl-7.83.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 264 100 264 0 0 318 0 --:--:-- --:--:-- --:--:-- 318
100 269 100 269 0 0 179 0 0:00:01 0:00:01 --:--:-- 0
100 25179 0 25179 0 0 9782 0 --:--:-- 0:00:02 --:--:-- 9410k
$ curl --connect-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1,*.curie.fr'
* Uses proxy env variable https_proxy == 'http://www-cache:3128'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 192.168.160.61:3128...
* Connected to www-cache (192.168.160.61) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to omnipathdb.org:443
> CONNECT omnipathdb.org:443 HTTP/1.1
> Host: omnipathdb.org:443
> User-Agent: curl/7.70.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Date: Fri, 27 May 2022 15:57:03 GMT
< Proxy-Connection: Keep-Alive
< Via: 1.1 stentor.curie.net
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CONNECT phase completed!
* CONNECT phase completed!
0 0 0 0 0 0 0 0 --:--:-- 0:01:02 --:--:-- 0* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to omnipathdb.org:443
0 0 0 0 0 0 0 0 --:--:-- 0:01:03 --:--:-- 0
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to omnipathdb.org:443
I don't remember having this problem with curl but I think I had a similar error message when using R.
Sorry for the late reply! After updating curl
we get a different error, confirming that the old curl
was an issue (edit: no, actually with our server there was no cypher issue, it's the same proxy issue since we lifted the timeout to 600). The new curl shows compatible cyphers (e.g. TLS_AES_128_GCM_SHA256 with TLS 1.3 should work), still the TLS handshake is not successful. It is possible that the firewall swallows some packets interrupting the connection, or curl attempts to use wrong protocol when communicating to the proxy. I think you can try a number of things:
SSL_ERROR_SYSCALL
) and try various solutions, there is plenty of discussion around the internethttp_proxy
or https_proxy
environment variables have wrong value. Above we see Uses proxy env variable https_proxy == 'http://www-cache:3128'
. What if you set it to https
?progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
https_proxy="https://www-cache:3128" curl -vvv --connection-timeout 600 -o 'progeny.tsv' -L $progeny_url
-vvv
flagswget
, aria2
?progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
wget $progeny_url
aria2c $progeny_url
openssl
itself establish a TLS connection?openssl s_client -connect omnipathdb.org:443
curl
download from another https sites?curl -vvvLO 'https://sabnzbd.org/tests/internetspeed/20MB.bin'
Here is a very similar, likely identical issue, and they went through similar points like us here: https://curl.se/mail/archive-2020-01/0021.html
Also here, no solution, but some interesting ideas in the comments: https://stackoverflow.com/questions/66090119/openssl-ssl-connect-ssl-error-syscall-in-connection-to-www-google-com443
Here some suggest to try the --proxy-tlsv1
and --proxy-insecure
options: https://bbs.archlinux.org/viewtopic.php?id=241982
progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
curl -vvv --connection-timeout 600 --proxy-tlsv1 -o 'progeny.tsv' -L $progeny_url
curl -vvv --connection-timeout 600 --proxy-insecure -o 'progeny.tsv' -L $progeny_url
Hi, I'm using the devel version and my network is behind a proxy. Running the following command gives an error