16EAGLE / getSpatialData

An R package 📦 making it easy to query, preview, download and preprocess multiple kinds of spatial data 🛰 via R. All beta.
https://jakob.schwalb-willmann.de/getSpatialData
GNU General Public License v3.0
303 stars 64 forks source link

Error in .get(paste0(getOption("gSD.api")$espa, "/order/", x) #71

Open MLezamaValdes opened 4 years ago

MLezamaValdes commented 4 years ago

Hi, I'm getting the following error whenever I try to access the records using check_availability(), getLandsat_data(), get_data() or order_data():

Error in .get(paste0(getOption("gSD.api")$espa, "/order/", x), getOption("gSD.usgs_user"), : Internal Server Error (HTTP 500). Failed to process request.

I am logged in via login_earthdata() and login_USGS() and getting the query is not an issue, only downloading the query content.

aoi.zip library(rgdal) aoi <- readOGR("Levy_MDV.shp") l8proj <- crs("+proj=utm +zone=57 +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs ") aoiutm <- spTransform(aoi, l8proj) set_aoi(aoiutm)

q <- getLandsat_records(time_range = c("2019-01-01", "2019-01-01"), name = "LANDSAT_8_C1", aoi=get_aoi()) q <- q[q$level=="bt",]

availability <- check_availability(q) getLandsat_data(records=q) get_data(q) order_data(q)

Thank you very much for taking a look!

susewuse commented 4 years ago

Same problem here too! Thanks

mkovac03 commented 4 years ago

I have also encountered the same issue!

mkovac03 commented 4 years ago

Creating a new ESPA account seemed to have solved the issue for me.

However, order_data() appears to make an individual order for each selected scene. As a result of that, after ordering each available SR images over my study area, I woke up with 6842 new items in my inbox ...

Moreover, get_data() doesn't start the download for the already completed scenes, even though they each have a separate order ID and are ready for download, but instead waits for all of them to complete.

Needless to say, this may be suboptimal for some, especially when working with large time series datasets.

Is there a way to you could look into this?

jakobjassmann commented 4 years ago

I am also experiencing this issue, which - I think - is related to the (solved) issue #66

For me the error is caused by previous order requests that are non-conformant to the expected naming pattern. These order ids do not have a time-stamp and therefore cannot be parsed by the relevant sections of code (e.g. line 63 onwards in check_availability.R). This then results in NA being introduced into the order_ids vector, which throws up the error when the script tries to look up the order details.

Here is a reproducible example with some of real order ids from my order history:

order_ids <- c(
   "espa-j.assmann@bios.au.dk-10092020-105452-283", 
   "espa-j.assmann@bios.au.dk-10092020-105455-277", 
   "espa-P100d8jamwhst", 
   "espa-P100d8itjpydn", 
   "espa-P100d8jjfs3gp", 
   "espa-P100d0uwribel", 
   "espa-P100d0uqcp4c3", 
   "espa-j.assmann@bios.au.dk-0102006128399"
   )
order_dates <- lapply(order_ids, function(x) strptime(strsplit(x, "-")[[1]][3], format = "%m%d%Y"))
order_ids <- order_ids[sapply(order_dates, function(x) difftime(Sys.time(), x, units = "days")) <= 7]

order_ids
#[1] "espa-j.assmann@bios.au.dk-10092020-105452-283" "espa-j.assmann@bios.au.dk-10092020-105455-277"
#[3] NA                                              NA                                             
#[5] NA                                              NA                                             
#[7] NA      

I was able to fix the issue locally by adding a simple order_ids <- na.omit(order_ids) before the if(length(order_ids) > 0) call (line 67).

@16EAGLE Maybe this could be a fix to be integrated into the code?