search

phenology / springtime

Spatiotemporal phenology research with interpretable models
https://springtime.readthedocs.io
Apache License 2.0
3 stars 2 forks source link

Added Appeears* classes #177

Closed sverhoeven closed 1 year ago

sverhoeven commented 1 year ago

Try to use https://appeears.earthdatacloud.nasa.gov/ application to download modis.

Refs #130

Application can also download daymet and many other products.

sverhoeven commented 1 year ago

Have submitted 500 point and germany area task they are still processing for 33% 3.4h and 69% 2.5h resp.

sverhoeven commented 1 year ago
The germany task completed in 5.5 hours with a MCD15A2H.061_500m_aid0001.nc file of 2.9Gb ```shell netcdf MCD15A2H.061_500m_aid0001 { dimensions: time = 667 ; lat = 1520 ; lon = 1874 ; variables: byte crs ; crs:grid_mapping_name = "latitude_longitude" ; crs:_CoordinateAxisTypes = "GeoX GeoY" ; crs:spatial_ref = "GEOGCS[\"unknown\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AXIS[\"Longitude\",EAST],AXIS[\"Latitude\",NORTH]]" ; crs:epsg_code = 4326LL ; crs:horizontal_datum_name = "WGS84" ; crs:semi_major_axis = 6378137LL ; crs:inverse_flattening = 298.257223563 ; crs:longitude_of_prime_meridian = 0. ; int time(time) ; time:standard_name = "time" ; time:axis = "T" ; time:calendar = "julian" ; time:units = "days since 2000-01-01 00:00:00" ; double lat(lat) ; lat:standard_name = "latitude" ; lat:units = "degrees_north" ; lat:_CoordinateAxisType = "GeoY" ; lat:axis = "Y" ; double lon(lon) ; lon:standard_name = "longitude" ; lon:units = "degrees_east" ; lon:_CoordinateAxisType = "GeoX" ; lon:axis = "X" ; short Fpar_500m(time, lat, lon) ; Fpar_500m:_FillValue = 255s ; Fpar_500m:coordinates = "time lat lon" ; Fpar_500m:grid_mapping = "crs" ; Fpar_500m:valid_min = 0s ; Fpar_500m:valid_max = 100s ; Fpar_500m:scale_factor_err = 0. ; Fpar_500m:add_offset_err = 0. ; Fpar_500m:calibrated_nt = 21 ; Fpar_500m:long_name = "MOD15A2H MODIS/Terra Gridded 500M FPAR (8-day composite)" ; Fpar_500m:units = "Percent" ; Fpar_500m:MOD15A2_FILLVALUE_DOC = "MOD15A2 FILL VALUE LEGEND\n255 = _Fillvalue, assigned when:\n * the MOD09GA suf. reflectance for channel VIS, NIR was assigned its _Fillvalue, or\n * land cover pixel itself was assigned _Fillvalus 255 or 254.\n254 = land cover assigned as perennial salt or inland fresh water.\n253 = land cover assigned as barren, sparse vegetation (rock, tundra, desert.)\n252 = land cover assigned as perennial snow, ice.\n251 = land cover assigned as \"permanent\" wetlands/inundated marshlands.\n250 = land cover assigned as urban/built-up.\n249 = land cover assigned as \"unclassified\" or not able to determine.\n" ; Fpar_500m:scale_factor = 0.01 ; Fpar_500m:add_offset = 0. ; short Lai_500m(time, lat, lon) ; Lai_500m:_FillValue = 255s ; Lai_500m:coordinates = "time lat lon" ; Lai_500m:grid_mapping = "crs" ; Lai_500m:valid_min = 0s ; Lai_500m:valid_max = 100s ; Lai_500m:scale_factor_err = 0. ; Lai_500m:add_offset_err = 0. ; Lai_500m:calibrated_nt = 21 ; Lai_500m:long_name = "MOD15A2H MODIS/Terra Gridded 500M Leaf Area Index LAI (8-day composite)" ; Lai_500m:units = "m^2/m^2" ; Lai_500m:MOD15A2_FILLVALUE_DOC = "MOD15A2 FILL VALUE LEGEND\n255 = _Fillvalue, assigned when:\n * the MOD09GA suf. reflectance for channel VIS, NIR was assigned its _Fillvalue, or\n * land cover pixel itself was assigned _Fillvalus 255 or 254.\n254 = land cover assigned as perennial salt or inland fresh water.\n253 = land cover assigned as barren, sparse vegetation (rock, tundra, desert.)\n252 = land cover assigned as perennial snow, ice.\n251 = land cover assigned as \"permanent\" wetlands/inundated marshlands.\n250 = land cover assigned as urban/built-up.\n249 = land cover assigned as \"unclassified\" or not able to determine.\n" ; Lai_500m:scale_factor = 0.1 ; Lai_500m:add_offset = 0. ; short FparLai_QC(time, lat, lon) ; FparLai_QC:_FillValue = 255s ; FparLai_QC:coordinates = "time lat lon" ; FparLai_QC:grid_mapping = "crs" ; FparLai_QC:valid_min = 0s ; FparLai_QC:valid_max = 254s ; FparLai_QC:long_name = "MOD15A2H MODIS/Terra+Aqua QC for FPAR and LAI (8-day composite)" ; FparLai_QC:units = "class-flag" ; FparLai_QC:FparLai_QC_DOC = "\nFparLai_QC 5 BITFIELDS IN 8 BITWORD\nMODLAND_QC START 0 END 0 VALIDS 2\nMODLAND_QC 0 = Good Quality (main algorithm with or without saturation)\nMODLAND_QC 1 = Other Quality (back-up algorithm or fill value)\nSENSOR START 1 END 1 VALIDS 2\nSENSOR 0 = Terra\nSENSOR 1 = Aqua\nDEADDETECTOR START 2 END 2 VALIDS 2\nDEADDETECTOR 0 = Detectors apparently fine for up to 50% of channels 1,2\nDEADDETECTOR 1 = Dead detectors caused >50% adjacent detector retrieval\nCLOUDSTATE START 3 END 4 VALIDS 4 (this inherited from Aggregate_QC bits {0,1} cloud state)\nCLOUDSTATE 00 = 0 Significant clouds NOT present (clear)\nCLOUDSTATE 01 = 1 Significant clouds WERE present\nCLOUDSTATE 10 = 2 Mixed cloud present on pixel\nCLOUDSTATE 11 = 3 Cloud state not defined,assumed clear\nSCF_QC START 5 END 7 VALIDS 5\nSCF_QC 000=0 Main (RT) algorithm used, best result possible (no saturation)\nSCF_QC 001=1 Main (RT) algorithm used, saturation occured. Good, very usable.\nSCF_QC 010=2 Main algorithm failed due to bad geometry, empirical algorithm used\nSCF_QC 011=3 Main algorithm faild due to problems other than geometry, empirical algorithm used\nSCF_QC 100=4 Pixel not produced at all, value coudn\'t be retrieved (possible reasons: bad L1B data, unusable MOD09GA data)\n" ; short FparExtra_QC(time, lat, lon) ; FparExtra_QC:_FillValue = 255s ; FparExtra_QC:coordinates = "time lat lon" ; FparExtra_QC:grid_mapping = "crs" ; FparExtra_QC:valid_min = 0s ; FparExtra_QC:valid_max = 254s ; FparExtra_QC:long_name = "MOD15A2H MODIS/Terra+Aqua pass-through QC for FPAR and LAI (8-day composite)" ; FparExtra_QC:units = "class-flag" ; FparExtra_QC:FparExtra_QC_DOC = "\nFparExtra_QC 6 BITFIELDS IN 8 BITWORD\nLANDSEA PASS-THROUGH START 0 END 1 VALIDS 4\nLANDSEA 00 = 0 LAND AggrQC(3,5)values{001}\nLANDSEA 01 = 1 SHORE AggrQC(3,5)values{000,010,100}\nLANDSEA 10 = 2 FRESHWATER AggrQC(3,5)values{011,101}\nLANDSEA 11 = 3 OCEAN AggrQC(3,5)values{110,111}\nSNOW_ICE (from Aggregate_QC bits) START 2 END 2 VALIDS 2\nSNOW_ICE 0 = No snow/ice detected\nSNOW_ICE 1 = Snow/ice were detected\nAEROSOL START 3 END 3 VALIDS 2\nAEROSOL 0 = No or low atmospheric aerosol levels detected\nAEROSOL 1 = Average or high aerosol levels detected\nCIRRUS (from Aggregate_QC bits {8,9} ) START 4 END 4 VALIDS 2\nCIRRUS 0 = No cirrus detected\nCIRRUS 1 = Cirrus was detected\nINTERNAL_CLOUD_MASK START 5 END 5 VALIDS 2\nINTERNAL_CLOUD_MASK 0 = No clouds\nINTERNAL_CLOUD_MASK 1 = Clouds were detected\nCLOUD_SHADOW START 6 END 6 VALIDS 2\nCLOUD_SHADOW 0 = No cloud shadow detected\nCLOUD_SHADOW 1 = Cloud shadow detected\nSCF_BIOME_MASK START 7 END 7 VALIDS 2\nSCF_BIOME_MASK 0 = Biome outside interval <1,4>\nSCF_BIOME_MASK 1 = Biome in interval <1,4>\n" ; // global attributes: :title = "MCD15A2H.061 for aid0001" ; :Conventions = "CF-1.6" ; :institution = "Land Processes Distributed Active Archive Center (LP DAAC)" ; :source = "AppEEARS v3.32" ; :references = "See README.md" ; :history = "See README.md" ; ``` ```json { "geo": { "type": "FeatureCollection", "features": [ { "type": "Feature", "geometry": { "type": "Polygon", "coordinates": [ [ [ 7.083984, 47.282959 ], [ 7.083984, 53.611084 ], [ 14.888672, 53.611084 ], [ 14.888672, 47.282959 ], [ 7.083984, 47.282959 ] ] ] }, "properties": {} } ], "fileName": "User-Drawn-Polygon" }, "dates": [ { "endDate": "12-31", "recurring": true, "startDate": "01-01", "yearRange": [ 2000, 2016 ] } ], "layers": [ { "layer": "Fpar_500m", "product": "MCD15A2H.061" }, { "layer": "Lai_500m", "product": "MCD15A2H.061" } ], "output": { "format": { "type": "netcdf4" }, "projection": "geographic" } } ```
Peter9192 commented 1 year ago

Took the liberty of addressing my own comments. Recipe is running now, but it takes a long time...

Peter9192 commented 1 year ago

East Frankfurt finished :-)

Peter9192 commented 1 year ago

Germany also finished. Now waiting on points

Peter9192 commented 1 year ago

OK, all downloads finished (it took about 10 hours, some of which queued) but test recipe failed with

raise ValueError(f"Indexes have overlapping values: {overlap}")
ValueError: Indexes have overlapping values: Index(['Fpar_500m_1', 'Fpar_500m_2', 'Fpar_500m_3', 'Fpar_500m_4',
       'Fpar_500m_5', 'Fpar_500m_6', 'Fpar_500m_7', 'Fpar_500m_8',
       'Fpar_500m_9', 'Fpar_500m_10', 'Fpar_500m_11', 'Fpar_500m_12',
       'Lai_500m_1', 'Lai_500m_2', 'Lai_500m_3', 'Lai_500m_4', 'Lai_500m_5',
       'Lai_500m_6', 'Lai_500m_7', 'Lai_500m_8', 'Lai_500m_9', 'Lai_500m_10',
       'Lai_500m_11', 'Lai_500m_12'],
      dtype='object')

I think this is expected behaviour, users should pick only one flavour for their recipe. In the (very) long run, we might want to add a better way to deal with this, but for now I'm happy to see that the downloads work as expected.