search

ua-snap / alfresco_inputs

scripts to build the input maps for the ALFRESCO Fire Model
2 stars 1 forks source link

FINALIZE TEM DOWNSCALING #29

Open EarthScientist opened 8 years ago

EarthScientist commented 8 years ago

DOWNSCALING COMPLETION CHECKLIST:

EarthScientist commented 8 years ago

Code that needs to be built out:

EarthScientist commented 8 years ago

small issue exists in the clouds downscaled data... Not sure exactly what is happening but we are getting values outside of 0-100 which is obviously incorrect and not possible.

Check the scaling on the values to see if I am making an error as to what the units really are and what scale they are in. This downscaling takes some time to run and we should check the tas and hur as well.

This may be a deal breaker for a monday delivery, but there is little I can do. This is a lot of processing and in some of these it is more complex than I might have liked. This will be sorted out and the proper and correct outputs will be distributed to the end users.

EarthScientist commented 8 years ago

the values of the cloud cover percentage are the same across the historical CRU TS data and the AR5 modeled data. they are in the range of 0.0-100.0

Not sure what is happening in the script to get such out of bounds errors.... there are no negs so how would we end up with negs?

_should we be taking the absolute value?_

EarthScientist commented 8 years ago

_cru_ts31 clouds issue has been isolated and solved._ Ended up it was grabbing the wrong group of climatology files.... It is now fixed and will be run as soon as we have some room on one of the processing servers.

EarthScientist commented 8 years ago

running vapor pressure conversion over the cru_ts3.1 data, run tests on data after series is produced before AR5 production run which will be a long process as it is lots of data.

EarthScientist commented 8 years ago

List of number of files in output AR5 directory:

The magic number of a perfectly output dataset is: 6432

PATH COUNT COMPLETE
'CCSM4/cld/downscaled' 6432 OK
'CCSM4/hur/downscaled' 0
'CCSM4/tas/downscaled' 6432 OK
'GFDL-CM3/cld/downscaled' 6302
'GFDL-CM3/tas/downscaled' 980 Running...
'GISS-E2-R/cld/downscaled' 6432 OK
'GISS-E2-R/hur/downscaled' 0
'GISS-E2-R/tas/downscaled' 6432 OK
'IPSL-CM5A-LR/cld/downscaled' 6432 OK
'IPSL-CM5A-LR/hur/downscaled' 0
'IPSL-CM5A-LR/tas/downscaled' 6432 OK
'MRI-CGCM3/cld/downscaled' 5292
'MRI-CGCM3/hur/downscaled' 0
'MRI-CGCM3/tas/downscaled' 6432 OK

Relative Humidity is quite obviously broken and I am spending time now fixing it. There are others in the list that also need to be properly output and I am focusing on these outputs now.

EarthScientist commented 8 years ago

relative humidity is now running over all AR5 models and scenarios. The issue was a lack of passing a plev arg which is needed to reduce the dimensionality of the netcdf since it has pressure levels associated with it. we are taking plev=1000.

Error Below:

_AR5 historical hur 1860-01 downscaled GISS_ hur_giss_historical_error

As can be seen there is an issue where there are a large number of pixels outside the 0-100 acceptable range and after following Stephanie McAfee's advice from the last run, we are to set those values to 99.0. This is causing for a noticeable change in large swaths of the map, as are seen in the more wet areas of the AOI above.

_AR5 rcp45 hur 2093-12 downscaled GISS_ hur_giss_rcp45_example

Seen here there is not a noticeable error, which is interesting to say the least. I am not sure what is causing this but it will need to be solved before releasing the data to the SEL group.

EarthScientist commented 8 years ago

I have found another error in how I was calculating the proportional anomalies. I have fixed this error along with some other ones that were causing for much slower processing times and am running them now.

The error was only in the AR5 outputs not the CRU TS3.1.

EarthScientist commented 8 years ago

Right now we are running from ~1hr per model/scenario/variable

we have 5 models, 5 scenarios, 4 variables currently needing running. We are 18 hours into the run and there is little we can do to speed it up. waiting...