Closed amcilraithRRC closed 3 years ago
I can't seem to reproduce this on Linux 20.04 nor on Windows 10. Travis CI is also fine but weirdly appveyor is failing with the same message (I've also received an email from someone regarding the same issue).
@boshek Can you reproduce this on your machine?
I'm wondering if it's a company/government network issue. @amcilraithRRC are you on a personal computer or work computer?
Hi Steffi; I am using a college computer on my home network, Shaw as a provider and all firewalls are turned off. Will try loading up weathercan on another computer to see what happens. It is interesting that the function to download stationIDs works but the one for downloading data does not. Will keep you posted. Alex
From: Steffi LaZerte [mailto:notifications@github.com] Sent: Friday, October 9, 2020 09:57 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
I can't seem to reproduce this on Linux 20.04 nor on Windows 10. Travis CI is also fine but weirdly appveyor is failing with the same message (I've also received an email from someone regarding the same issue).
@boshekhttps://github.com/boshek Can you reproduce this on your machine?
I'm wondering if it's a company/government network issue. @amcilraithRRChttps://github.com/amcilraithRRC are you on a personal computer or work computer?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-706230945, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEDOVSCRLYGN4O26FSLSJ4QFTANCNFSM4SJO3R2A.
Hi Steffi; Tried on another computer that I use for research and it does work. Notes:
On the college laptop, I upgraded R and R-studio to the latest versions (R is 4.0.2) since my version was too old to properly load weathercan and its dependencies.
On the other machine, I am running R 3.5.2. Not sure if this is relevant. Alex
From: Steffi LaZerte [mailto:notifications@github.com] Sent: Friday, October 9, 2020 09:57 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
I can't seem to reproduce this on Linux 20.04 nor on Windows 10. Travis CI is also fine but weirdly appveyor is failing with the same message (I've also received an email from someone regarding the same issue).
@boshekhttps://github.com/boshek Can you reproduce this on your machine?
I'm wondering if it's a company/government network issue. @amcilraithRRChttps://github.com/amcilraithRRC are you on a personal computer or work computer?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-706230945, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEDOVSCRLYGN4O26FSLSJ4QFTANCNFSM4SJO3R2A.
Thanks for trouble shooting! For the laptop that does work, is it a personal laptop or another college/work laptop? I'm trying to figure out if this is a permissions problem with curl, which might be funny on institution computers.
The computer that does work is a personal machine. Is there a way that I can debug to see what the problem is? I teach programming and networks, so I am interested in doing some digging. ;)
From: Steffi LaZerte [mailto:notifications@github.com] Sent: Friday, October 9, 2020 11:27 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
Thanks for trouble shooting! For the laptop that does work, is it a personal laptop or another college/work laptop? I'm trying to figure out if this is a permissions problem with curl, which might be funny on institution computers.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-706278416, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEBAMNY7ZEILJ4XTKU3SJ42WRANCNFSM4SJO3R2A.
Here the url that your query is ultimately accessing: https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10
One step is to see if that even works in the browser.
That'd be great! These kinds of problems just make me wanna cry!
As @boshek pointed out, a good place to start is simply with the link in your browser. It should initiate a csv download.
Also, here is a simple example with expected output of what weathercan does under the hood using the httr package (which uses the curl package). If the url doesn't work then none of this will, but if url does work, then the problem might be in httr/curl (I'm not sure where to go exactly from there, but let me know if you have any questions).
library(httr)
q <- list(format = "csv", stationID = 51097, timeframe = 1, submit = 'Download+Data', Year = 2020, Month = 10)
data <- GET(url = getOption("weathercan.urls.weather"), query = q)
data
# Response [https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10]
# Date: 2020-10-09 17:42
# Status: 200
# Content-Type: application/force-download
# Size: 83.4 kB
# <BINARY BODY>
data$request$url
# [1] "https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10"
content(data, as = "text")
# No encoding supplied: defaulting to UTF-8.
# [1] "\"Longitude (x)\",\"Latitude (y)\",\"Station Name\",\"Climate ID\",\"Date/Time\",\"Year\",\"Month\",\"Day\",\"Time\",\"Temp (°C)\",\"Temp Flag\",\"Dew Point Temp (°C)\",\"Dew Point Temp Flag\",\"Rel Hum (%)\",\"Rel Hum Flag\",\"Wind Dir (10s deg)\",\"Wind Dir Flag\",\"Wind Spd (km/h)\",\"Wind Spd Flag\",\"Visibility (km)\",\"Visibility Flag\",\"Stn Press (kPa)\",\"Stn Press Flag\",\"Hmdx\",\"Hmdx Flag\",\"Wind Chill\",\"Wind Chill Flag
This URL works (Firefox 81.0.1); the problem appears to be in how R is trying to access the site. When I had initially accessed data I used this sort of link from the EC website and it worked. I will look at info in Steffi’s email.
From: Sam Albers [mailto:notifications@github.com] Sent: Friday, October 9, 2020 12:42 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
Here the url that your query is ultimately accessing: https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10
One step is to see if that even works in the browser.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-706315671, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEALO5LW5KG5IPDWFWTSJ5DOZANCNFSM4SJO3R2A.
Same script, different results from two computers.
Output from "good” computer:
library(httr)
Warning message:
package ‘httr’ was built under R version 3.5.3
q <- list(format = "csv", stationID = 51097, timeframe = 1, submit = 'Download+Data', Year = 2020, Month = 10)
data <- GET(url = getOption("weathercan.urls.weather"), query = q)
data
Date: 2020-10-09 17:56
Status: 200
Content-Type: application/force-download
Size: 83.4 kB
read.csv
doesn't use the R curl library (pretty sure). Does this work?
read.csv("https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=csv&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10")
[cid:image002.png@01D69E3D.995115B0]
On problem computer, it does not work.
From: Sam Albers [mailto:notifications@github.com] Sent: Friday, October 9, 2020 13:08 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
read.csv doesn't use the R curl library (pretty sure). Does this work?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-706328082, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEHNDFKJLI4TXWQQ2IDSJ5GQDANCNFSM4SJO3R2A.
@amcilraithRRC can you also give the error messages please? Is it the same error message?
I attached an image in the last email; maybe it did not come thru. Will attach as jpg. For some reason I am not being able to copy the console text on this “problem” machine. Alex
From: Sam Albers [mailto:notifications@github.com] Sent: Friday, October 9, 2020 13:18 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
@amcilraithRRChttps://github.com/amcilraithRRC can you also give the error messages please? Is it the same error message?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-706332559, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEAETB4DEYPW2VXOHSLSJ5HVTANCNFSM4SJO3R2A.
If you want to include an image, I think you'll need to go to GitHub issues directly and add it by dragging/dropping or pasting them into the window here.
Added image of error - Alex
Tried a different URL and I did get a response from the server to a simple http_get:
library(httr) myURL <- "http://www.rrc.ca" GET(url=myURL)
Argh I'm at a bit of a loss!
I will play with it a bit as i find inspiration. Would be nice to know root cause in case any one else runs into it. Thanks for all the help so far!
Thank you for putting in the effort, it's really nice to have the assistance :)
Really just throwing some mud at the wall now but does this in browser work: https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=txt&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10
or this directly with {curl}:
curl::curl_fetch_memory('https://climate.weather.gc.ca/climate_data/bulk_data_e.html?format=txt&stationID=51097&timeframe=1&submit=Download%2BData&Year=2020&Month=10')
Hello, I think I am finding some bread crumbs, I captured the packets being sent from my computer to the weathercan server. It appears that the weathercan server is rejecting the TLS handshake as it comes from R and then immediately resets the connection. I will try capturing the same connection attempt on the other computer and see what the difference(s) are.
I'm excited to report that I think I've figured out a fix for those users experiencing problems.
Delving into the curl
package's vignette for Proxies and Certificates on Windows Networks, I found that there are two SSL backends for Windows: OpenSSL and Windows Secure Channel. By default curl
uses the Windows Secure Channel which is supposed to be best, but doesn't quite have the full functionality of OpenSSL.
My best guess is that something changed on the ECCC server that required a functionality not provided by Windows Secure Channel. When I forced my windows testing platform (AppVeyor) to use OpenSSL, all my tests cleared.
If you would like to try this, you'll have to add the following to your .Renviron file (which is loaded during R's startup):
CURL_SSL_BACKEND=openssl
If you're unsure how to edit your .Renviron file, you can use the usethis
package to find/create and open it for you:
install.packages("usethis")
usethis::edit_r_environ()
Disclaimer: I don't think there are security drawbacks for doing this, but I'm not a network specialist by any stretch!
If you decide to give this a try, let me know how it goes!
I'm getting the same error in a corporate environment. The .Renviron
change did not work.
Error: Problem with
mutate()input
html. x schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). i Input
htmlis
purrr::map(...). Run
rlang::last_error()` to see where the error occurred.
R> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1
Matrix products: default
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages: [1] stats graphics grDevices datasets utils methods base
other attached packages:
[1] plotly_4.9.2.9000 kableExtra_1.2.1 weathercan_0.4.0.9999
[4] dplyr_1.0.2 rmdformats_0.3.7 knitr_1.30
[7] ggplot2_3.3.2.9000 extrafont_0.17
loaded via a namespace (and not attached):
[1] tidyselect_1.1.0 xfun_0.18 purrr_0.3.4 lattice_0.20-41
[5] colorspace_1.4-1 vctrs_0.3.4 generics_0.0.2 htmltools_0.5.0
[9] viridisLite_0.3.0 yaml_2.2.1 rlang_0.4.8 pillar_1.4.6
[13] glue_1.4.2 withr_2.3.0 sp_1.4-4 lifecycle_0.2.0
[17] stringr_1.4.0 munsell_0.5.0 gtable_0.3.0 rvest_0.3.6
[21] htmlwidgets_1.5.2 evaluate_0.14 memoise_1.1.0 curl_4.3
[25] fansi_0.4.1 Rttf2pt1_1.3.8 highr_0.8 Rcpp_1.0.5
[29] scales_1.1.1 webshot_0.5.2 jsonlite_1.7.1 digest_0.6.25
[33] stringi_1.5.3 bookdown_0.21 grid_4.0.2 cli_2.1.0
[37] tools_4.0.2 magrittr_1.5 lazyeval_0.2.2 tibble_3.0.3
[41] crayon_1.3.4 extrafontdb_1.0 tidyr_1.1.2 pkgconfig_2.0.3
[45] ellipsis_0.3.1 data.table_1.13.0 xml2_1.3.2 lubridate_1.7.9
[49] assertthat_0.2.1 rmarkdown_2.4 httr_1.4.2 rstudioapi_0.11
[53] R6_2.4.1 compiler_4.0.2 `
Just noticed that you are running Windows 7. Any chance you can try on a Windows 10 machine on the same network?
Drat, the change worked for three others else affected (although I'm not sure about @amcilraithRRC).
Can you try running: curl::curl_version()
and share the output with us. This should help figure out which SSL backends you have and which one you're using.
Do you use a company proxy?
- Can you try running:
curl::curl_version()
and share the output with us. This should help figure out which SSL backends you have and which one you're using.
I just installed the latest curl
. Here's the output:
` R> curl::curl_version() $version [1] "7.64.1"
$ssl_version [1] "(OpenSSL/1.1.1a) Schannel"
$libz_version [1] "1.2.11"
$libssh_version [1] "libssh2/1.8.2"
$libidn_version [1] NA
$host [1] "x86_64-w64-mingw32"
$protocols [1] "dict" "file" "ftp" "ftps" "gopher" "http" "https" "imap" "imaps" [10] "ldap" "ldaps" "pop3" "pop3s" "rtsp" "scp" "sftp" "smtp" "smtps" [19] "telnet" "tftp"
$ipv6 [1] TRUE
$http2 [1] FALSE
$idn [1] TRUE`
Hi, Could it be a security certificate issue with climate.weater.gc.ca again (October 2019 issue)? E.
On Tue., Oct. 20, 2020, 11:05 Steffi LaZerte, notifications@github.com wrote:
Drat, the change worked for three others else affected (although I'm not sure about @amcilraithRRC https://github.com/amcilraithRRC).
1.
Can you try running: curl::curl_version() and share the output with us. This should help figure out which SSL backends you have and which one you're using. 2.
Do you use a company proxy?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ropensci/weathercan/issues/104#issuecomment-712877947, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGNA6HUYQ75VNVBUSQRX6GLSLWKMFANCNFSM4SJO3R2A .
Just noticed that you are running Windows 7. Any chance you can try on a Windows 10 machine on the same network?
Windows 10 has no issues. Unfortunately, I can only work in Win 7.
@ebourlon It could be, unfortunately I'm at a complete loss now.
@tspeidel-suncor You may be better luck asking on the curl page?
Sorry 😞
Hi; I did find out that the machine that does not work for me is Windows 8.1. The working one is Windows 10. Have been working on other projects, but will poke at this one again soon. Alex
From: Steffi LaZerte [mailto:notifications@github.com] Sent: Wednesday, October 21, 2020 15:50 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
@ebourlonhttps://github.com/ebourlon It could be, unfortunately I'm at a complete loss now.
@tspeidel-suncorhttps://github.com/tspeidel-suncor You may be better luck asking on the curlhttps://jeroen.cran.dev/curl/ page?
Sorry 😞
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-713869422, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGECW6GOZ4VACVEJB4XTSL5CQPANCNFSM4SJO3R2A.
As a workaround for SSL problems on old versions of windows, you can switch to the openssl backend by setting
CURL_SSL_BACKEND=openssl
In your ~/.Renviron
file. See this vignette for more details.
We did suggest that as a possible solution, I think @tspeidel-suncor's problem is that this solution didn't work for them. Any reason why it wouldn't?
I think he didn't set it in the correct file, because the error suggests it is still using windows native ssl, not openssl.
@tspeidel-suncor can you run this in R:
writeLines("CURL_SSL_BACKEND=openssl", "~/.Renviron")
Make sure you restart R after that, because the change gets activated when the curl package is first loaded.
I think he didn't set it in the correct file, because the error suggests it is still using windows native ssl, not openssl.
@tspeidel-suncor can you run this in R:
writeLines("CURL_SSL_BACKEND=openssl", "~/.Renviron")
Make sure you restart R after that, because the change gets activated when the curl package is first loaded.
That's correct, I did try adding openssl to my .Renviron
to no avail. I tried again using the writelines()
, but still get ther error:
> kam <- weather_dl(station_ids = 51423, start = "2018-02-01", end = "2018-04-15")
Error: Problem with `mutate()` input `html`.
x schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed).
i Input `html` is `purrr::map…`.
Run `rlang::last_error()` to see where the error occurred.
I tried it without reloading R and it failed. After reloading R, it works perfectly! Many thanks, Alex (Using Windows 8.1)
From: Thomas Speidel [mailto:notifications@github.com] Sent: Monday, October 26, 2020 10:28 To: ropensci/weathercan weathercan@noreply.github.com Cc: Alex McIlraith AMcIlraith@RRC.CA; Mention mention@noreply.github.com Subject: Re: [ropensci/weathercan] Error when trying to access weather data in R ----> SEC_E_ILLEGAL_MESSAGE (0x80090326) (#104)
CAUTION: This email originates from outside RRC. Please verify the sender and always use caution with any requests, links or attached documents.
I think he didn't set it in the correct file, because the error suggests it is still using windows native ssl, not openssl.
@tspeidel-suncorhttps://github.com/tspeidel-suncor can you run this in R:
writeLines("CURL_SSL_BACKEND=openssl", "~/.Renviron")
Make sure you restart R after that, because the change gets activated when the curl package is first loaded.
That's correct, I did try adding openssl to my .Renviron to no avail. I tried again using the writelines(), but still get ther error:
kam <- weather_dl(station_ids = 51423, start = "2018-02-01", end = "2018-04-15")
Error: Problem with mutate()
input html
.
x schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed).
i Input html
is purrr::map…
.
Run rlang::last_error()
to see where the error occurred.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/weathercan/issues/104#issuecomment-716623108, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARJZGEA2CXPAVR6K77BPE6DSMWIOLANCNFSM4SJO3R2A.
Expected Behavior
Weather data should download in RCurrent Behavior
install.packages("weathercan")
install.packages("lutz")
install.packages("sf")
library("weathercan") library("lutz") library("sf") s <- stations_dl() weather_dl(station_ids = 51097, start = "2020-10-01", end = "2020-10-08",interval="hour") Error: Problem with
mutate()
inputhtml
. x schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). i Inputhtml
ispurrr::map(...)
. Runrlang::last_error()
to see where the error occurred.Steps to Reproduce (for bugs)
See above
Possible Solution
Context
Unable to use weathercan
Your Environment
Windows 10