search

bluegreen-labs / daymetr

An R Interface to the Daymet Web Services
http://bluegreen-labs.github.io/daymetr/
Other
30 stars 17 forks source link

ncss download inconsistency #32

Closed khufkens closed 5 years ago

khufkens commented 5 years ago

Downloads with the default query: download_daymet_ncss(path = "~")

Which queries: https://thredds.daac.ornl.gov/thredds/ncss/ornldaac/1328/1980/daymet_v3_tmin_1980_na.nc4?var=lat&var=lon&var=tmin&north=34&west=-82&east=-81.75&south=33.75&time_start=1980-01-01T12:00:00Z&time_end=1980-12-31T12:00:00Z&timeStride=1&accept=netcdf

Will results in different downloads for unknown reasons!

At times a query will download 54K: Creating a subset of the Daymet data be patient, this might take a while! Downloading DAYMET subset: year: 1980; product: tmin Downloading: 54 kB

Which only seems to contain one pixel of data (gdalinfo)

Driver: netCDF/Network Common Data Format
Files: tmin_daily_1980_ncss.nc
Size is 512, 512
Coordinate System is `'
Metadata:
  NC_GLOBAL#citation=Please see http://daymet.ornl.gov/ for current Daymet data citation information
  NC_GLOBAL#Conventions=CF-1.6
  NC_GLOBAL#geospatial_lat_max=12.01033018985965
  NC_GLOBAL#geospatial_lat_min=11.93425580314189
  NC_GLOBAL#geospatial_lon_max=-79.48915510825471
  NC_GLOBAL#geospatial_lon_min=-79.76852761774354
  NC_GLOBAL#History=Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)
Original Dataset = /daymet/V3/CFMosaic/1980/daymet_v3_tmin_1980_na.nc4; Translation Date = 2018-11-20T17:21:42.020Z
  NC_GLOBAL#references=Please see http://daymet.ornl.gov/ for current information on Daymet references
  NC_GLOBAL#source=Daymet Software Version 3.0
  NC_GLOBAL#start_year=1980
  NC_GLOBAL#Version_data=Daymet Data Version 3.0
  NC_GLOBAL#Version_software=Daymet Software Version 3.0
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lat
  SUBDATASET_1_DESC=[1x34] latitude (32-bit floating-point)
  SUBDATASET_2_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lon
  SUBDATASET_2_DESC=[1x34] longitude (32-bit floating-point)
  SUBDATASET_3_NAME=NETCDF:"tmin_daily_1980_ncss.nc":tmin
  SUBDATASET_3_DESC=[365x1x34] tmin (32-bit floating-point)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

A short time later it will download 1.3MB: Creating a subset of the Daymet data be patient, this might take a while! Downloading DAYMET subset: year: 1980; product: tmin Downloading: 1.3 MB

Which contains data (gdalinfo)

Driver: netCDF/Network Common Data Format
Files: tmin_daily_1980_ncss.nc
Size is 512, 512
Coordinate System is `'
Metadata:
  NC_GLOBAL#citation=Please see http://daymet.ornl.gov/ for current Daymet data citation information
  NC_GLOBAL#Conventions=CF-1.6
  NC_GLOBAL#geospatial_lat_max=34.05315791533045
  NC_GLOBAL#geospatial_lat_min=33.70520494123185
  NC_GLOBAL#geospatial_lon_max=-81.68213717040001
  NC_GLOBAL#geospatial_lon_min=-82.06525821633383
  NC_GLOBAL#History=Translated to CF-1.0 Conventions by Netcdf-Java CDM (CFGridWriter2)
Original Dataset = /daymet/V3/CFMosaic/1980/daymet_v3_tmin_1980_na.nc4; Translation Date = 2018-11-20T17:24:15.970Z
  NC_GLOBAL#references=Please see http://daymet.ornl.gov/ for current information on Daymet references
  NC_GLOBAL#source=Daymet Software Version 3.0
  NC_GLOBAL#start_year=1980
  NC_GLOBAL#Version_data=Daymet Data Version 3.0
  NC_GLOBAL#Version_software=Daymet Software Version 3.0
Subdatasets:
  SUBDATASET_1_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lat
  SUBDATASET_1_DESC=[32x28] latitude (32-bit floating-point)
  SUBDATASET_2_NAME=NETCDF:"tmin_daily_1980_ncss.nc":lon
  SUBDATASET_2_DESC=[32x28] longitude (32-bit floating-point)
  SUBDATASET_3_NAME=NETCDF:"tmin_daily_1980_ncss.nc":tmin
  SUBDATASET_3_DESC=[365x32x28] tmin (32-bit floating-point)
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  512.0)
Upper Right (  512.0,    0.0)
Lower Right (  512.0,  512.0)
Center      (  256.0,  256.0)

I'm unsure if this is an httr issue or a Daymet issue.

khufkens commented 5 years ago

This issue is still unfixed, or returned for some reason (refresh of the system).

The coordinate system is missing in the ncss queries, which results in errors with raster reads, failing the unit checks. The required coordinates system should read something like this.

I blank out the unit tests until this gets resolved after the gov. shutdown.

+proj=lcc +lat_1=25 +lat_2=60 +lat_0=42.5 +lon_0=-100 +x_0=0 +y_0=0 +a=6378137 +b=6356752.314706705 +units=m +no_defs

khufkens commented 5 years ago

Things are stable again it seems.

We have applied a new configuration on storage, which we hope is addressing the data corruption issue. If you are still seeing corrupt data, we have another solution to this on-the-ready but we have not implemented on the public server. That is, to throttle requests when requests are too high. If you feel corrupted files are still being delivered, we might see if you can do some testing to see if the throttling soln corrects the problem.

The issue of needing specifically formatted coordinate reference system information in the netCDF files.

Just today as a result of your message and the user contacting us, I noticed a subtle inconsistency in an attribute of the lcc coordinate reference system in our files. I think it’s due to an update to IDL that went un-noticed.

Can you tell me if one of these files works differently than the other in your DaymetR code?

daymet_v3_vp_2017_puertorico.nc4 vs daymet_v3_vp_2015_puertorico.nc4

khufkens commented 5 years ago

fixed