search

ropensci / rnoaa

R interface to many NOAA data APIs
https://docs.ropensci.org/rnoaa
Other
328 stars 85 forks source link

homr() gives Error: Bad gateway (HTTP 502) #384

Closed windnick closed 3 years ago

windnick commented 3 years ago

Recent changes to NOAA server cause HTTP 502 error. Blog posts suggest this can be avoided by forcing HTTP version 1.0

Session Info ```r R version 4.0.2 (2020-06-22) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19041) Matrix products: default locale: [1] LC_COLLATE=English_United Kingdom.1252 [2] LC_CTYPE=English_United Kingdom.1252 [3] LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C [5] LC_TIME=English_United Kingdom.1252 attached base packages: [1] stats graphics grDevices utils datasets methods base other attached packages: [1] rnoaa_1.1.0 RevoUtils_11.0.2 RevoUtilsMath_11.0.0 loaded via a namespace (and not attached): [1] Rcpp_1.0.5 xml2_1.3.2 magrittr_1.5 rappdirs_0.3.1 [5] tidyselect_1.1.0 munsell_0.5.0 colorspace_1.4-1 R6_2.3.0 [9] rlang_0.4.6 hoardr_0.5.2 dplyr_1.0.0 grid_4.0.2 [13] gtable_0.3.0 ellipsis_0.3.1 digest_0.6.25 httpcode_0.3.0 [17] tibble_3.0.2 lifecycle_0.2.0 crayon_1.3.4 gridExtra_2.3 [21] tidyr_1.1.0 purrr_0.3.4 ggplot2_3.3.2 vctrs_0.3.1 [25] triebeard_0.3.0 curl_3.3 crul_0.9.0 glue_1.4.1 [29] compiler_4.0.2 pillar_1.4.6 urltools_1.7.3 generics_0.0.2 [33] scales_1.1.1 XML_3.99-0.4 lubridate_1.7.9 jsonlite_1.5 [37] pkgconfig_2.0.3 ```
sckott commented 3 years ago

I see you're using an older version of rnoaa, try upating to latest rnoaa on cran and try again - if that still doens't work share a reproducible example - see https://reprex.tidyverse.org/ if you aren't sure how to do it

windnick commented 3 years ago

Argh! Default ROpen repos is 7 months out of date! Installed rnoaa_1.3.0 from GitHub, but same HTTP 502 error. Presume one or more dependencies are also out of date. Which?

library(rnoaa) sessionInfo() R version 4.0.2 (2020-06-22) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale: [1] LC_COLLATE=English_United Kingdom.1252 [2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] rnoaa_1.3.0 RevoUtils_11.0.2 RevoUtilsMath_11.0.0

loaded via a namespace (and not attached): [1] Rcpp_1.0.5 xml2_1.3.2 magrittr_1.5 rappdirs_0.3.1
[5] tidyselect_1.1.0 munsell_0.5.0 colorspace_1.4-1 R6_2.3.0
[9] rlang_0.4.6 hoardr_0.5.2 dplyr_1.0.0 tools_4.0.2
[13] grid_4.0.2 gtable_0.3.0 ellipsis_0.3.1 digest_0.6.25
[17] httpcode_0.3.0 tibble_3.0.2 lifecycle_0.2.0 crayon_1.3.4
[21] gridExtra_2.3 tidyr_1.1.0 purrr_0.3.4 ggplot2_3.3.2
[25] vctrs_0.3.1 curl_3.3 crul_0.9.0 glue_1.4.1
[29] compiler_4.0.2 pillar_1.4.6 generics_0.0.2 scales_1.1.1
[33] XML_3.99-0.4 lubridate_1.7.9 jsonlite_1.5 pkgconfig_2.0.3

x<-homr(quid="ICAO:KIAD") Error: Bad Gateway (HTTP 502)

sckott commented 3 years ago

Lets see if it's a general problem with your setup. do any of these work?

library(rnoaa)
ghcnd(stationid = "AGE00147704")
lcd(station = "01338099999", year = 2015)
bsw(date = "2017-10-01")
windnick commented 3 years ago

Thanks for your help. Response to your code is:

library(rnoaa) ghcnd(stationid = "AGE00147704") using cached file: C:\Users\swift\AppData\Local\Cache/R/noaa_ghcnd/AGE00147704.dly date created (size, mb): 2021-02-05 10:33:37 (0.487) A tibble: 963 x 128 id year month element VALUE1 MFLAG1 QFLAG1 SFLAG1 VALUE2 MFLAG2 QFLAG2

1 AGE0~ 1909 11 TMAX NA " " " " " " NA " " " " 2 AGE0~ 1909 11 TMIN NA " " " " " " NA " " " " 3 AGE0~ 1909 11 PRCP NA " " " " " " NA " " " " 4 AGE0~ 1909 12 TMAX 170 " " " " "E" 160 " " " " 5 AGE0~ 1909 12 TMIN 90 " " " " "E" 100 " " " " 6 AGE0~ 1909 12 PRCP 24 " " " " "E" 0 " " " " 7 AGE0~ 1910 1 TMAX 140 " " " " "E" 120 " " " " 8 AGE0~ 1910 1 TMIN 100 " " " " "E" 80 " " " " 9 AGE0~ 1910 1 PRCP 82 " " " " "E" 176 " " " " 10 AGE0~ 1910 2 TMAX 110 " " " " "E" 140 " " " " ... with 953 more rows, and 117 more variables: SFLAG2 , VALUE3 , MFLAG3 , QFLAG3 , SFLAG3 , VALUE4 , MFLAG4 , QFLAG4 , SFLAG4 , VALUE5 , MFLAG5 , QFLAG5 , SFLAG5 , VALUE6 , MFLAG6 , QFLAG6 , SFLAG6 , VALUE7 , MFLAG7 , QFLAG7 , SFLAG7 , VALUE8 , MFLAG8 , QFLAG8 , SFLAG8 , VALUE9 , MFLAG9 , QFLAG9 , SFLAG9 , VALUE10 , MFLAG10 , QFLAG10 , SFLAG10 , VALUE11 , MFLAG11 , QFLAG11 , SFLAG11 , VALUE12 , MFLAG12 , QFLAG12 , SFLAG12 , VALUE13 , MFLAG13 , QFLAG13 , SFLAG13 , VALUE14 , MFLAG14 , QFLAG14 , SFLAG14 , VALUE15 , MFLAG15 , QFLAG15 , SFLAG15 , VALUE16 , MFLAG16 , QFLAG16 , SFLAG16 , VALUE17 , MFLAG17 , QFLAG17 , SFLAG17 , VALUE18 , MFLAG18 , QFLAG18 , SFLAG18 , VALUE19 , MFLAG19 , QFLAG19 , SFLAG19 , VALUE20 , MFLAG20 , QFLAG20 , SFLAG20 , VALUE21 , MFLAG21 , QFLAG21 , SFLAG21 , VALUE22 , MFLAG22 , QFLAG22 , SFLAG22 , VALUE23 , MFLAG23 , QFLAG23 , SFLAG23 , VALUE24 , MFLAG24 , QFLAG24 , SFLAG24 , VALUE25 , MFLAG25 , QFLAG25 , SFLAG25 , VALUE26 , MFLAG26 , QFLAG26 , SFLAG26 , VALUE27 , MFLAG27 , QFLAG27 , ... Warning message: `...` is not empty.

We detected these problematic arguments:

These dots only exist to allow future extensions and should be empty. Did you misspecify an argument?

lcd(station = "01338099999", year = 2015) using cached file: C:\Users\swift\AppData\Local\Cache/R/noaa_lcd/2015_01338099999.csv date created (size, mb): 2021-02-05 10:34:08 (4.177) A tibble: 8,096 x 121 station date latitude longitude elevation name report_type source

1 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 2 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 3 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 4 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 5 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 6 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 7 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 8 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 9 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 10 1.34e9 2015~ 61.2 6.65 51 VANG~ FM-12 4 ... with 8,086 more rows, and 113 more variables: hourlyaltimetersetting , hourlydewpointtemperature , hourlydrybulbtemperature , hourlyprecipitation , hourlypresentweathertype , hourlypressurechange , hourlypressuretendency , hourlyrelativehumidity , hourlyskyconditions , hourlysealevelpressure , hourlystationpressure , hourlyvisibility , hourlywetbulbtemperature , hourlywinddirection , hourlywindgustspeed , hourlywindspeed , sunrise , sunset , dailyaveragedewpointtemperature , dailyaveragedrybulbtemperature , dailyaveragerelativehumidity , dailyaveragesealevelpressure , dailyaveragestationpressure , dailyaveragewetbulbtemperature , dailyaveragewindspeed , dailycoolingdegreedays , dailydeparturefromnormalaveragetemperature , dailyheatingdegreedays , dailymaximumdrybulbtemperature , dailyminimumdrybulbtemperature , dailypeakwinddirection , dailypeakwindspeed , dailyprecipitation , dailysnowdepth , dailysnowfall , dailysustainedwinddirection , dailysustainedwindspeed , dailyweather , monthlyaveragerh , monthlydayswithgt001precip , monthlydayswithgt010precip , monthlydayswithgt32temp , monthlydayswithgt90temp , monthlydayswithlt0temp , monthlydayswithlt32temp , monthlydeparturefromnormalaveragetemperature , monthlydeparturefromnormalcoolingdegreedays , monthlydeparturefromnormalminimumtemperature , monthlydeparturefromnormalprecipitation , monthlydewpointtemperature , monthlygreatestprecip , monthlygreatestprecipdate , monthlygreatestsnowdepth , monthlygreatestsnowdepthdate , monthlygreatestsnowfall , monthlygreatestsnowfalldate , monthlymaxsealevelpressurevalue , monthlymaxsealevelpressurevaluedate , monthlymaxsealevelpressurevaluetime , monthlymaximumtemperature , monthlymeantemperature , monthlyminsealevelpressurevalue , monthlyminsealevelpressurevaluedate , monthlyminsealevelpressurevaluetime , monthlyminimumtemperature , monthlysealevelpressure , monthlystationpressure , monthlytotalliquidprecipitation , monthlytotalsnowfall , monthlywetbulb , awnd , cdsd , cldd , dsnw , hdsd , htdd , normalscoolingdegreeday , normalsheatingdegreeday , shortdurationenddate005 , shortdurationenddate010 , shortdurationenddate015 , shortdurationenddate020 , shortdurationenddate030 , shortdurationenddate045 , shortdurationenddate060 , shortdurationenddate080 , shortdurationenddate100 , shortdurationenddate120 , shortdurationenddate150 , shortdurationenddate180 , shortdurationprecipitationvalue005 , shortdurationprecipitationvalue010 , shortdurationprecipitationvalue015 , shortdurationprecipitationvalue020 , shortdurationprecipitationvalue030 , shortdurationprecipitationvalue045 , shortdurationprecipitationvalue060 , shortdurationprecipitationvalue080 , shortdurationprecipitationvalue100 , shortdurationprecipitationvalue120 , ... Warning message: `...` is not empty.

We detected these problematic arguments:

These dots only exist to allow future extensions and should be empty. Did you misspecify an argument?

bsw(date = "2017-10-01") Error: Could not resolve host: eclipse.ncdc.noaa.gov

homr(quid="ICAO:KIAD") Error: Bad Gateway (HTTP 502) homr(quid="COOP:448903") Error: Bad Gateway (HTTP 502) homr(quid="WBAN:93738") Error: Bad Gateway (HTTP 502)

sessionInfo() R version 4.0.2 (2020-06-22) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19041)

Matrix products: default

locale: [1] LC_COLLATE=English_United Kingdom.1252 [2] LC_CTYPE=English_United Kingdom.1252
[3] LC_MONETARY=English_United Kingdom.1252 [4] LC_NUMERIC=C
[5] LC_TIME=English_United Kingdom.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] rnoaa_1.3.0 RevoUtils_11.0.2 RevoUtilsMath_11.0.0

loaded via a namespace (and not attached): [1] Rcpp_1.0.5 pillar_1.4.6 compiler_4.0.2 tools_4.0.2
[5] digest_0.6.25 jsonlite_1.5 lubridate_1.7.9 lifecycle_0.2.0
[9] tibble_3.0.2 gtable_0.3.0 pkgconfig_2.0.3 rlang_0.4.6
[13] cli_2.0.2 crul_0.9.0 curl_3.3 gridExtra_2.3
[17] dplyr_1.0.0 xml2_1.3.2 generics_0.0.2 vctrs_0.3.1
[21] rappdirs_0.3.1 triebeard_0.3.0 grid_4.0.2 tidyselect_1.1.0 [25] glue_1.4.1 httpcode_0.3.0 data.table_1.12.8 R6_2.3.0
[29] fansi_0.4.1 XML_3.99-0.4 ggplot2_3.3.2 purrr_0.3.4
[33] tidyr_1.1.0 hoardr_0.5.2 magrittr_1.5 scales_1.1.1
[37] urltools_1.7.3 ellipsis_0.3.1 assertthat_0.2.1 colorspace_1.4-1 [41] utf8_1.1.4 munsell_0.5.0 crayon_1.3.4

sckott commented 3 years ago

Okay, i think it's just that some NOAA services are very unreliable; sometimes they work and sometimes they do not work. If homr never works for you that suggests a different problem, but if it sometimes works then we can't do much about that since that's a NOAA problem that we can't solve

windnick commented 3 years ago

Well, I guess the only solution when it's Someone Else's Problem is the write one's own code. The following will retrieve most useful ASOS station metadata from NOAA HOMR server.

library(stringr) library(lubridate)

metaASOS=function(qid) {#----------------------------------- local functions ------------------------------------- numextract <- function(string){str_extract(string, "\-\d+\.\d*")}

return n characters after expr from str

Seg=function(expr,n,str)
{if (length(str)>1) str<-paste(str,collapse="") y<-gregexpr(expr,str) i<-unlist(y)+nchar(expr) ni<-length(i) s<-character() for (j in c(1:ni)) {s<-append(s,substr(str,i[j],i[j]+n-1))} return(s) }

return integer within n characters of expr in str

Id<-function(expr,n,str)  
  {s<-Seg(expr,n,str)[1]
   s<-gsub("[^0-9]", "", s)
 return(as.numeric(s))
}
# return first character string within n characters of expr in str
Str<-function(expr,n,str) 
    {s<-Seg(expr,n,str)[1]
   i<-gregexpr('[A-Z]',s,ignore.case=TRUE)[[1]][1] 
   s<-substr(s,i,n)
   i<-gregexpr('\"',s)[[1]][1]
   return(substr(s,1,i-1))
  }

------------------------------------ get homr page for qid ----------------------------------

url<-paste('http://ncdc.noaa.gov/homr/services/station/search?qid=',qid,'&date=all',sep="") options(warn=-1) homr<-readLines(url) options(warn=0)

------------------------------------ get current station name, etc ---------------------------

expr<-'\",\"names\":\[\{\"name\"' StnName<-Str(expr,100,homr) county<-Str('county',100,homr) s<-Seg('stateProvinces',100,homr); state<-Str('stateProvince',100,s)

---------------------------------------- UTC offset --------------------------------------

s<-Seg('utcOffsets',25,homr); Hoffset<-as.numeric(numextract(s))[1]

------------------------------------ get station Id codes ------------------------------

NCDC<-Id('ncdcStnId',20,homr) COOP<-Id('\"COOP\",\"id\":\"',10,homr) WBAN<-Id('\"WBAN\",\"id\":\"',10,homr) WMO<-Id('\"WMO\",\"id\":\"',10,homr) ICAO<-Seg('\"ICAO\",\"id\":\"',4,homr)[1] FAA<-substr(ICAO,2,4)

---------------------------------------- get latitudes and longitudes ------------------

expr<-'latitude_dec\":\"' s<-Seg(expr,100,homr) s<-s[grepl("beginDate",s)] # latitude strings with begin date latitudes<-as.numeric(numextract(substr(s,1,15))) dates<-Seg('beginDate\":\"',10,s) latDates<-make_datetime(as.numeric(substr(dates,1,4)), as.numeric(substr(dates,6,7)),as.numeric(substr(dates,9,10))) expr<-'longitude_dec\":\"' s<-Seg(expr,100,homr) s<-s[grepl("beginDate",s)] # longitude strings with begin date longitudes<-as.numeric(numextract(substr(s,1,15))) dates<-Seg('beginDate\":\"',10,s) longDates<-make_datetime(as.numeric(substr(dates,1,4)), as.numeric(substr(dates,6,7)),as.numeric(substr(dates,9,10)))

--------------------------------------- get Sonic install date -----------------------

expr<-'ANEMSONIC'; s<-Seg(expr,50,homr) expr<-'beginDate'; s<-Seg(expr,15,s) s<-substr(tail(s,n=1),4,13) # last one found is earliest sonicDate<-make_datetime(as.numeric(substr(s,1,4)), as.numeric(substr(s,6,7)),as.numeric(substr(s,9,10)))

--------------------------------------- get elevations --------------------------------

expr<-'elevationType'; s<-Seg(expr,120,homr) expr<-'GROUND'; s<-Seg(expr,120,s) expr<-'elevationMeters'; s<-Seg(expr,80,s) elevations<-as.numeric(numextract(substr(s,1,15))) dates<-Seg('beginDate\":\"',10,s) elevDates<-make_datetime(as.numeric(substr(dates,1,4)), as.numeric(substr(dates,6,7)),as.numeric(substr(dates,9,10)))

-------------------------------------- compile data list -------------------------------

return(list(StnName=StnName,county=county,state=state,utcOffset=Hoffset, NCDC=NCDC,COOP=COOP,WBAN=WBAN,WMO=WMO,ICAO=ICAO,FAA=FAA,latitudes=latitudes, latDates=latDates,longitudes=longitudes,longDates=longDates,elevations=elevations,elevDates=elevDates, sonicDate=sonicDate)) }