search

saezlab / decoupleR

R package to infer biological activities from omics data using a collection of methods.
https://saezlab.github.io/decoupleR/
GNU General Public License v3.0
183 stars 24 forks source link

Error in open.connection(con, "rb") Timeout was reached #57

Closed kaizen89 closed 2 years ago

kaizen89 commented 2 years ago

Hi, I'm using the devel version and my network is behind a proxy. Running the following command gives an error

net <- get_progeny(organism = 'human', top = 100)
Error in open.connection(con, "rb") : 
  Timeout was reached: [omnipathdb.org] Operation timed out after 10010 milliseconds with 0 out of 0 bytes received
deeenes commented 2 years ago

Hi,

Have you tried it many times? Timeout is often matter of luck. In very bad networks it's possible that 10 sec is never enough to establish the connection (often due to DNS issues).

Increasing the timeout would require to set the curl option CONNECTTIMEOUT. Unfortunately the curl package provides API to set it only on the curl_handle objects. And the readr package has absolutely no API to set any curl options. And the curl package doesn't support the R options API. It makes our job quite difficult when we try to address this in OmnipathR. I'm still thinking about a solution, but let me know if repeated attempts work.

Best,

Denes

deeenes commented 2 years ago

I added an option to set curl connection timeout, you can try it by updating OmnipathR from the latest commit in its master branch. The default value of this option (omnipath.connect_timeout) is the same as the default of curl (10 seconds). To raise it to 60 seconds:

library(OmnipathR)
options(omnipath.connect_timeout = 60)

I can't really test if it works as I don't know a server which consistently gives a connect timeout, and the curl R package lacks the API to inspect handlers.

kaizen89 commented 2 years ago

Thanks for your reactivity, unfortunately after many attempts, setting omnipath.connect_timeout still same error message.

deeenes commented 2 years ago

Hi,

To confirm that indeed the long connection time is responsible for the failure, you could run curl from the command line (bash) with its default and an increased timeout:

progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'

curl -vvv -o 'progeny.tsv' -L $progeny_url
curl --connection-timeout 60 -vvv -o 'progeny-to60.tsv' -L $progeny_url

We expect the first to fail, and the second maybe to succeed. If the second one fails too due to a connect timeout, it means in my opinion, that your connection just stalls and even an infinite timeout couldn't fix the problem. It can happen if let's say the DNS never replies, gives a wrong response, or a firewall swallows all the response, and under many further weird network conditions.

Also my quick hack with the timeout option in OmnipathR broke its behaviour under certain conditions, hence I suggest to update it again to version 3.5.2 where I fixed these issues.

kaizen89 commented 2 years ago

Both failed unfortunately

$ curl --connect-timeout 60 -vvv -o 'progeny-to60.tsv' -L $progeny_url
*   Trying 192.168.160.60...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to (nil) (192.168.160.60) port 3128 (#0)
* Establish HTTP proxy tunnel to omnipathdb.org:443
> CONNECT omnipathdb.org:443 HTTP/1.1
> Host: omnipathdb.org:443
> User-Agent: curl/7.52.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< Date: Wed, 25 May 2022 08:44:07 GMT
< Proxy-Connection: Keep-Alive
< Via: 1.1 stentor.curie.net
< 
* Proxy replied OK to CONNECT request
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /home/salmon/anaconda2/ssl/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:00:59 --:--:--     0* Operation timed out after 60614 milliseconds with 0 out of 0 bytes received
* Curl_http_done: called premature == 1
  0     0    0     0    0     0      0      0 --:--:--  0:01:00 --:--:--     0
* Closing connection 0
curl: (28) Operation timed out after 60614 milliseconds with 0 out of 0 bytes received
deeenes commented 2 years ago

But it looks like communication has just started before it timed out? This is very extreme, but maybe you could try with even longer connect timeout:

progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
curl --connection-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
kaizen89 commented 2 years ago 
curl --connect-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
*   Trying 192.168.160.61...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to (nil) (192.168.160.61) port 3128 (#0)
* Establish HTTP proxy tunnel to omnipathdb.org:443
> CONNECT omnipathdb.org:443 HTTP/1.1
> Host: omnipathdb.org:443
> User-Agent: curl/7.52.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< Date: Wed, 25 May 2022 10:40:50 GMT
< Proxy-Connection: Keep-Alive
< Via: 1.1 stentor.curie.net
< 
* Proxy replied OK to CONNECT request
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /home/salmon/anaconda2/ssl/cacert.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:--  0:01:02 --:--:--     0* Unknown SSL protocol error in connection to omnipathdb.org:443 
* Curl_http_done: called premature == 1
  0     0    0     0    0     0      0      0 --:--:--  0:01:03 --:--:--     0
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to omnipathdb.org:443
deeenes commented 2 years ago

This is now a different error. The reasons behind are not clear, but likely a missing certificate on your computer or the incompatibility between the cipher suites supported by your computer and our server. These happen most often due to old client side software. Your curl is 6 years old, I would recommend to update it. Before, it would be interesting to check the supported protocols and ciphers:

curl --version

curl https://www.howsmyssl.com/a/check

curl -o 'curl-7.52.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'

Then you could check again everything: the supported standards, accessing OmniPath server, and would be nice to test some other server too. Btw, have you ever experienced similar issue with any other webpage, especially if accessed programmatically?

curl --version

curl https://www.howsmyssl.com/a/check

curl -o 'curl-7.83.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'

progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
curl --connection-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
kaizen89 commented 2 years ago

Before update

$ curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2l zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets HTTPS-proxy 
$ curl https://www.howsmyssl.com/a/check
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option
$curl -o 'curl-7.52.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   264  100   264    0     0    142      0  0:00:01  0:00:01 --:--:--   142
100   269  100   269    0     0     70      0  0:00:03  0:00:03 --:--:--   197
100 35438    0 35438    0     0   5814      0 --:--:--  0:00:06 --:--:-- 22760

After update

$ curl --version
curl 7.70.0 (x86_64-pc-linux-gnu) libcurl/7.70.0 OpenSSL/1.1.1 zlib/1.2.11
Release-Date: 2020-04-29
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets
$ curl https://www.howsmyssl.com/a/check
{"given_cipher_suites":["TLS_AES_256_GCM_SHA384","TLS_CHACHA20_POLY1305_SHA256","TLS_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_EMPTY_RENEGOTIATION_INFO_SCSV"],"ephemeral_keys_supported":true,"session_ticket_supported":false,"tls_compression_supported":false,"unknown_cipher_suite_supported":false,"beast_vuln":false,"able_to_detect_n_minus_one_splitting":false,"insecure_cipher_suites":{},"tls_version":"TLS 1.3","rating":"Probably Okay"}
$ curl -o 'curl-7.83.1-ssltest.html' -L 'https://www.ssllabs.com/ssltest/viewMyClient.html'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   264  100   264    0     0    318      0 --:--:-- --:--:-- --:--:--   318
100   269  100   269    0     0    179      0  0:00:01  0:00:01 --:--:--     0
100 25179    0 25179    0     0   9782      0 --:--:--  0:00:02 --:--:-- 9410k
$ curl --connect-timeout 600 -vvv -o 'progeny-to600.tsv' -L $progeny_url
* Uses proxy env variable no_proxy == 'localhost,127.0.0.0/8,::1,*.curie.fr'
* Uses proxy env variable https_proxy == 'http://www-cache:3128'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.160.61:3128...
* Connected to www-cache (192.168.160.61) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to omnipathdb.org:443
> CONNECT omnipathdb.org:443 HTTP/1.1
> Host: omnipathdb.org:443
> User-Agent: curl/7.70.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< Date: Fri, 27 May 2022 15:57:03 GMT
< Proxy-Connection: Keep-Alive
< Via: 1.1 stentor.curie.net
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* CONNECT phase completed!
* CONNECT phase completed!
  0     0    0     0    0     0      0      0 --:--:--  0:01:02 --:--:--     0* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to omnipathdb.org:443 
  0     0    0     0    0     0      0      0 --:--:--  0:01:03 --:--:--     0
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to omnipathdb.org:443

I don't remember having this problem with curl but I think I had a similar error message when using R.

deeenes commented 2 years ago

Sorry for the late reply! After updating curl we get a different error, confirming that the old curl was an issue (edit: no, actually with our server there was no cypher issue, it's the same proxy issue since we lifted the timeout to 600). The new curl shows compatible cyphers (e.g. TLS_AES_128_GCM_SHA256 with TLS 1.3 should work), still the TLS handshake is not successful. It is possible that the firewall swallows some packets interrupting the connection, or curl attempts to use wrong protocol when communicating to the proxy. I think you can try a number of things:

progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
https_proxy="https://www-cache:3128" curl -vvv --connection-timeout 600 -o 'progeny.tsv' -L $progeny_url
progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'
wget $progeny_url
aria2c $progeny_url
openssl s_client -connect omnipathdb.org:443
curl -vvvLO 'https://sabnzbd.org/tests/internetspeed/20MB.bin'

Here is a very similar, likely identical issue, and they went through similar points like us here: https://curl.se/mail/archive-2020-01/0021.html

Also here, no solution, but some interesting ideas in the comments: https://stackoverflow.com/questions/66090119/openssl-ssl-connect-ssl-error-syscall-in-connection-to-www-google-com443

Here some suggest to try the --proxy-tlsv1 and --proxy-insecure options: https://bbs.archlinux.org/viewtopic.php?id=241982

progeny_url='https://omnipathdb.org/annotations?resources=PROGENy'

curl -vvv --connection-timeout 600 --proxy-tlsv1 -o 'progeny.tsv' -L $progeny_url

curl -vvv --connection-timeout 600 --proxy-insecure -o 'progeny.tsv' -L $progeny_url