2015-09-02 GLM 2.0.4 Ice on and off comparison

lawinslow commented 9 years ago

Using GLM for ice-on and -off is the mda.lakes holy grail.

The Goal

Shuter et al 2013 used a simple multi-linear regression approach and got these stats

Ice on

-2.4 day bias (observed - predicted)
8.4 day precision (standard deviation(observed - predicted))
8.7 mean absolute error
Ice off
-3.0 day bias (observed - predicted)
7.4 day precision (standard deviation(observed - predicted))
5.2 mean absolute error

Interesting that they get better ice-off estimates. Probably the opposite of what we'll get. (edit: or not)

GLM v2.0.4 Results

Ice on

3.3 day bias (observed - predicted)
11.1 day precision (standard deviation(observed - predicted))
7.4 mean absolute error
Ice off
11.0 day bias (observed - predicted)
8.7 day precision (standard deviation(observed - predicted))
11.9 mean absolute error

Ok, we're getting there. It has been far worse. Ice on isn't great, but I have some anecdotal evidence from Dale that it can be fairly good. Ice off is all about the bias. If we can get rid of the 11 day bias, then we could get mean absolute error down to 6.5.

Also, here are some plots.

Note: below is ice off, ignore title

jordansread commented 9 years ago

looking very promising. NLDAS w/ wind bias removed?

lawinslow commented 9 years ago

Straight NLDAS. didn't correct wind, but I can

jordansread commented 9 years ago

just curious what the reference point is/was. We should discuss tomorrow, but this seems pretty positive. Another thing to consider is how error in either end (off vs on) propagates to the thermal metrics of interest.

lawinslow commented 9 years ago

Model tweaks

Ice/snow evaporation improved
Max albedo bumped from 0.7 to 0.8

GLM v2.0.4-AlbedoBump Results

Ice on

3.5 day bias (observed - predicted)
11.2 day precision (standard deviation(observed - predicted))
7.6 mean absolute error

(Interesting that these results change a little)

Ice off

1.1 day bias (observed - predicted)
8.1 day precision (standard deviation(observed - predicted))
6.2 mean absolute error

Significant improvement.

jordansread commented 9 years ago

wow, quite a jump in the I-O bias. Was the albedo or the evap change more important to that?

lawinslow commented 9 years ago

All albedo. I think I'm going to move forward switching to full runs instead of chained. What do you think?

jordansread commented 9 years ago

Yep. Does it require evap mass flux to be off?

lawinslow commented 9 years ago

Nope. Evap is now on, unlike before. Though granted, Winter Evap is pretty low. I didn't evaluate carefully, but I doubt it made a huge difference.

lawinslow commented 9 years ago

Note, earlier results were from a small increase to max snow albedo. Now improved implementation in GLM v3.1.0 - alpha. Using snow_albedo_factor = 1.1 we get similar results.

> cat('MAE of GLM Overall, on and off date/time')
MAE of GLM Overall, on and off date/time
> cat('Mean Absolute Errors\n')
Mean Absolute Errors
> mean(abs(both_ice$glm_doy - both_ice$doy))
[1] 7.011064
> mean(abs(both_ice[both_ice$on.off=='on',]$glm_doy - both_ice[both_ice$on.off=='on',]$doy))
[1] 7.639442
> mean(abs(both_ice[both_ice$on.off=='off',]$glm_doy - both_ice[both_ice$on.off=='off',]$doy))
[1] 6.542348
> 
> cat('STD of GLM Overall, on and off date/time')
STD of GLM Overall, on and off date/time
> cat('Standard Deviation\n')
Standard Deviation
> sd((both_ice$glm_doy - both_ice$doy))
[1] 9.586225
> sd((both_ice[both_ice$on.off=='on',]$glm_doy - both_ice[both_ice$on.off=='on',]$doy))
[1] 11.33798
> sd((both_ice[both_ice$on.off=='off',]$glm_doy - both_ice[both_ice$on.off=='off',]$doy))
[1] 8.038161
> 
> cat('Bias of GLM Overall, on and off date/time')
Bias of GLM Overall, on and off date/time
> cat('Bias\n')
Bias
> mean((both_ice$glm_doy - both_ice$doy))
[1] -3.23234
> mean((both_ice[both_ice$on.off=='on',]$glm_doy - both_ice[both_ice$on.off=='on',]$doy))
[1] -3.5
> mean((both_ice[both_ice$on.off=='off',]$glm_doy - both_ice[both_ice$on.off=='off',]$doy))
[1] -3.032689
> 
> cat('MAE of empirical model overall, on and off date/time')
MAE of empirical model overall, on and off date/time
> cat('Mean Absolute Errors\n')
Mean Absolute Errors
> mean(abs(both_ice$emp_doy - both_ice$doy))
[1] 6.18383
> mean(abs(both_ice[both_ice$on.off=='on',]$emp_doy - both_ice[both_ice$on.off=='on',]$doy))
[1] 6.709163
> mean(abs(both_ice[both_ice$on.off=='off',]$emp_doy - both_ice[both_ice$on.off=='off',]$doy))
[1] 5.791976
> 
> 
> cat('R^2')
R^2
> tmp = both_ice[both_ice$on.off == 'off', ]
> summary(lm(tmp$emp_doy ~ tmp$glm_doy))$r.squared
[1] 0.7406314
> summary(lm(tmp$doy ~ tmp$glm_doy))$r.squared
[1] 0.7147607
> summary(lm(tmp$doy ~ tmp$emp_doy))$r.squared
[1] 0.7732483

USGS-R / mda.lakes