Pulled_data error - Githubissues

acrunyon commented 2 years ago

I've adjusted the README.Rmd you provided to pull all years, all GCMs for 3 variables for a single grid cell (.Rmd attached). It looked to me like the input_times and input_variables were correct but I got the following error (screenshot attached). Any ideas what is going on?

Rmd I've been using

Untitled picture

CURL Error: Transferred a partial file Error in Rsx_nc4_get_vara_int: NetCDF: DAP failure Var: pr_BNU-ESM_r1i1p1_rcp85 Ndims: 3 Start: 0,444,511 Count: 34333,2,3 Error in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset, : C function Rsx_nc4_get_var_int returned error

gjknowlton commented 2 years ago

I had the same issue when running the example script that amber had above with the same error message:

CURL Error: Transferred a partial file Error in Rsx_nc4_get_vara_int: NetCDF: DAP failure Var: pr_CCSM4_r6i1p1_rcp85 Ndims: 3 Start: 0,444,511 Count: 34333,2,3 Error in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset, : C function Rsx_nc4_get_var_int returned error

ttuff commented 2 years ago

This is a runtime error. Data requests are limited to 500mb per call by the data provider. You are getting this error because you are requesting more than the maximum data allowed from one call. You will need to reduce the spatial extent or the temporal extent to get this below the maximum. When I reduce the spatial extent to a much smaller national park, the run time error goes away and I can get all the data. When I bump back up to the entire Yellowstone region, I need to stitch it together.

I will work on a few solutions to this problem, but for now, just request smaller data chunks each time.

First subset your time into a few groups. `time_min <- 38716 time_max <- 73048

input_times <- inputs$available_times %>% add_column(index = 0) %>% add_column(first_half = 0) %>% add_column(second_half = 0) input_times[which(inputs$available_times[,1] >= time_min & inputs$available_times[,1] <= time_max ),3] <- 1

med <- median(row_number(input_times[,3])) input_times[which(as.numeric(row.names(input_times)) <= med),4] <- 1 input_times[which(as.numeric(row.names(input_times)) > med),5] <- 1

head(input_times) tail(input_times)`

And then request those two subsets and stitch them back together.

`Pulled_data_sub1 <- inputs$src %>% hyper_filter(lat = lat <= c(pulled_bb[4]+0.05) & lat >= c(pulled_bb[2]-0.05)) %>% hyper_filter(lon = lon <= c(pulled_bb[3]+0.05) & lon >= c(pulled_bb[1]-0.05)) %>% hyper_filter(time = input_times[,4] == 1) %>% hyper_tibble(select_var = input_variables ) %>% st_as_sf(coords = c("lon", "lat"), crs = 4326, agr = "constant")

head(Pulled_data_sub1)

Pulled_data_sub2 <- inputs$src %>% hyper_filter(lat = lat <= c(pulled_bb[4]+0.05) & lat >= c(pulled_bb[2]-0.05)) %>% hyper_filter(lon = lon <= c(pulled_bb[3]+0.05) & lon >= c(pulled_bb[1]-0.05)) %>% hyper_filter(time = input_times[,5] == 1) %>% hyper_tibble(select_var = input_variables ) %>% st_as_sf(coords = c("lon", "lat"), crs = 4326, agr = "constant")

head(Pulled_data_sub2)

Pulled_data_stitch <- rbind(Pulled_data_sub1, Pulled_data_sub2) head(Pulled_data_stitch) tail(Pulled_data_stitch)`

I've added this to the new readme file for your reference and I will work on incorporating a parallel version to make this easier and faster.

Cheers, Ty

acrunyon commented 2 years ago

Hi Ty, Can we meet again sometime this week? I am only pulling a single grid cell, which is 98% of what we do. Like I said, for this to work for us, we have to be able to pull all years (1950-2100), GCMs, RCPs for tmax, tmin, precip, and preferably rhmin, rhmax. Before the rgdal updates broke the package, we were able to pull all of this data for a single grid cell in <2 hrs.

We have spent years building an infrastructure of analysis around these data, and have just built an R package based on it as well. I think it may help if I show you how we work with this data, so you have a better idea of our needs from this package.

From: Ty Tuff @.> Sent: Monday, November 1, 2021 11:45 To: earthlab/cft @.> Cc: Runyon, Amber N @.>; Author @.> Subject: [EXTERNAL] Re: [earthlab/cft] Pulled_data error (Issue #152)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.