search

USEPA / elevatr

An R package for accessing elevation data
Other
204 stars 26 forks source link

2%Error in curl::curl_fetch_memory(url, handle = handle) : Empty reply from server #29

Closed ridwanshittu closed 3 years ago

ridwanshittu commented 4 years ago

I got this error why trying to extract elevation for some locations. I have practised it before, it works but now the extraction stop at a point and return the above error

jhollist commented 4 years ago

Thanks for reporting this. Hard to say what the issue is without some details.

How large of an area are you trying to get data for and are you extracting point elevations or raster DEM'?

mafichman commented 4 years ago

I am having a similar error. I can grab elevations for 30-50ish x-y points, but right now I'm just operating using a small sample of my data to test the code. Ultimately I'm going to want to do this for thousands of points. It appears to time out or fail unpredictably - sometimes it can handle 30 points, sometimes it can't.

The data are simple x and y data, no NA values, no values outside the bounds of the continental US.

jhollist commented 4 years ago

For the data in the US are you using the "aws" or "epqs" for the src argument?

ridwanshittu commented 4 years ago

Thanks for reporting this. Hard to say what the issue is without some details.

How large of an area are you trying to get data for and are you extracting point elevations or raster DEM'?

Thank you for your response. The points area were large (distributed around the continent) and some of the points were not in the USA. I have noticed, the elevate r package works only with points in the USA.

Below is a sample from my code: rt contains about 5000 points rt <- read.csv("rtV_R-export_thinned.csv") rt <- rt[,2:3] head(rt) summary(rt) ll_prj <- "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs"

rt_sp <- sp::SpatialPoints(sp::coordinates(rt), proj4string = sp::CRS(ll_prj)) summary(rt_sp) plot(rt_sp) head(rt_sp)

df_elev_epqs <- get_elev_point(examp_df, prj = prj_dd, src = "epqs")

rtEl <- get_elev_point(rt_sp, prj = ll_prj, src = )

rtv <- get_aws_points(rt_sp, z = 5, units = c("meters", "feet"), ...)

rtv <- get_aws_points(rt )

mafichman commented 4 years ago

For the data in the US are you using the "aws" or "epqs" for the src argument?

I was using "epgs" but "aws" works like a charm. Thank you.

jhollist commented 4 years ago

Fantastic! Wonder if the epqs has implemented some rate limits... I'll take a look and see if I can can't recreate the issue with is.

On Thu, Feb 6, 2020 at 2:18 PM mafichman notifications@github.com wrote:

For the data in the US are you using the "aws" or "epqs" for the src argument?

I was using "epgs" but "aws" works like a charm. Thank you.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jhollist/elevatr/issues/29?email_source=notifications&email_token=ABJPYS6PYZ4FQFPOQVEUYFLRBRO7VA5CNFSM4KJST6Q2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELAOB5A#issuecomment-583065844, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJPYS2R2EJYX67UVORV733RBRO7VANCNFSM4KJST6QQ .

-- Jeffrey W. Hollister email: jeff.w.hollister@gmail.com cell: 401 556 4087 https://jwhollister.com

mafichman commented 4 years ago

The call would fail at seemingly random times, based on the verbose feedback. My guess is a single one would fail for one reason or another as it was going through the loop and then that would return a total failure rather than logging an NA and continuing on. Perhaps that's something that can be addressed - returning no data to an observation rather than having the loop close with an error.

johnrharley commented 4 years ago

I was also having the same timeout (empty reply) problem with my batch EPQS requests. I tried to slow down the request rate by modifying the Sys.sleep(1 / 100) argument in get_epqs(), but it still would throw up the fetch error, seemingly at random intervals. Similar to @mafichman it seemed to fail somewhat randomly.

I noticed that USGS offers bulk EPQS requests (https://viewer.nationalmap.gov/apps/bulk_pqs/) which are limited to 500 requests, but the 500 limit does not appear to apply to batch HTTP requests since I was able to get through more rows than that in a request before the failure. So I don't think there is a hard limit of requests/day or anything like that.

jhollist commented 3 years ago

@johnrharley , @rayscell, and @mafichman I am working on clearing out some issues.

Have you all still had issues with the larger number of points? I am testing 5000 points right now and have not had any issues yet (at ~90%). This is on the newest version of elevatr (0.4.0) which hit CRAN yesterday.

I'll circle back to this issue in about a week or two and will close if not continuing issues related to elevatr.

jresasco commented 3 years ago

I'm having the same error here and it seems idiosyncratic. Oddly, I was running the same code about a month ago without problems.

The following line will run successfully sometimes and other times return the error (Error in curl::curl_fetch_memory...) get_elev_point(locations = obs, units="meters", prj = "EPSG:4326") obs is 100 coordinates, no NAs

I'm looping through several hundred groups of observations and the error breaks the loop.

I'm using elevatr version 0.4.1 but had the same issue on verion 0.3.4 R version 4.0.2 (2020-06-22) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Catalina 10.15.7

Update: issues resolved when I specify src = "aws" 👍

jhollist commented 3 years ago

I am seeing simliar results. I am thinking it is on the API end. I'll make sure to look back at this next time I get some time to work on elevatr.

jhollist commented 3 years ago

All, this is on the API side. Occasionally the server sends back an empty response causing an error. I just pushed a fix to main, that re-tries the offending URL up to 5 times (this usually fixes the issue). If it is still empty after 5 tries, it returns NA, no longer errors, and also returns a warning that this has happened. This only applies to the epqs source. I think this closes this issue. If not, feel free to reopen.

Thank you all!