search

omegahat / RCurl

Other
29 stars 14 forks source link

url.exists() gives false negatives on Windows computers #43

Open mccarthy-m-g opened 4 years ago

mccarthy-m-g commented 4 years ago

For some reason, url.exists() has different behaviour between OSX, Linux, and Windows, where using the function in Windows falsely returns FALSE values on URLs that do exist. For example:

RCurl::url.exists(“https://www.netlify.com”)
#> FALSE

But on OSX or Linux it returns the correct value:

RCurl::url.exists(“https://www.netlify.com”)
#> TRUE

I’ve tested this on the following:

Some other users on Stackoverflow seem to have the same problem, and Windows seems to be the common denominator.

Do you have any solutions to this problem?

duncantl commented 4 years ago

Hi Michael I have a conjecture, but let's get some data. Can you issue the commands RCurl::curlVersion() and RCurl::url.exists("https://www.netlify.com", verbose = TRUE) and show us the output from the two commands, please? Thanks

mccarthy-m-g commented 4 years ago

Hi Duncan,

Here are the results from my local Windows and OSX computers. Both computers are using the most recent CRAN release of RCurl.

Windows

RCurl::curlVersion()
$age
[1] 3

$version
[1] "7.40.0"

$vesion_num
[1] 468992

$host
[1] "x86_64-pc-win32"

$features
      ssl      libz      ntlm asynchdns    spnego largefile       idn      sspi 
        4         8        16       128       256       512      1024      2048 

$ssl_version
[1] "OpenSSL/1.0.0o"

$ssl_version_num
[1] 0

$libz_version
[1] "1.2.8"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps" 
[10] "ldap"   "pop3"   "pop3s"  "rtmp"   "rtsp"   "scp"    "sftp"   "smtp"   "smtps" 
[19] "telnet" "tftp"  

$ares
[1] ""

$ares_num
[1] 0

$libidn
[1] ""
RCurl::url.exists("https://www.netlify.com", verbose = TRUE)
[1] FALSE

OSX

RCurl::curlVersion()
$age
[1] 3

$version
[1] "7.54.0"

$vesion_num
[1] 472576

$host
[1] "x86_64-apple-darwin18.0"

$features
     ipv6       ssl      libz      ntlm asynchdns    spnego largefile   ntlm_wb 
        1         4         8        16       128       256       512     32768 

$ssl_version
[1] "LibreSSL/2.6.5"

$ssl_version_num
[1] 0

$libz_version
[1] "1.2.11"

$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps"  "ldap"  
[11] "ldaps"  "pop3"   "pop3s"  "rtsp"   "smb"    "smbs"   "smtp"   "smtps"  "telnet" "tftp"  

$ares
[1] ""

$ares_num
[1] 0

$libidn
[1] ""
RCurl::url.exists("https://www.netlify.com", verbose = TRUE)
* Rebuilt URL to: https://www.netlify.com/
*   Trying 104.248.78.23...
* TCP_NODELAY set
* Connected to www.netlify.com (104.248.78.23) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=ca; L=San Francisco; O=Netlify, Inc; CN=*.netlify.com
*  start date: Jul  3 00:00:00 2019 GMT
*  expire date: Jul  7 12:00:00 2020 GMT
*  subjectAltName: host "www.netlify.com" matched cert's "*.netlify.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> HEAD / HTTP/1.1
Host: www.netlify.com
Accept: */*

< HTTP/1.1 200 OK
< Cache-Control: public, max-age=0, must-revalidate
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 19 May 2020 23:48:39 GMT
< Etag: "c0763cb5850d3ed513a7cf0a69525e01-ssl"
< Link: <https://www.netlify.com/>; rel="canonical"
< Strict-Transport-Security: max-age=31536000
< Age: 8
< Content-Length: 253471
< Connection: keep-alive
< Server: Netlify
< Set-Cookie: nf_ab=0.981820; expires=Wed, 19-May-2021 23:48:47 GMT; path=/
< X-NF-Request-ID: 09e89e2f-7399-4283-acf5-0974a20789f6-14993968
< 
* Connection #0 to host www.netlify.com left intact
[1] TRUE