run ensemble model on all UK

bogsnork / enPeatDepthModel

an open source peat depth model for England

1 stars 0 forks source link

run ensemble model on all UK #8

Open bogsnork opened 6 years ago

bogsnork commented 6 years ago

results from all models (except rf) uk1a

bogsnork commented 6 years ago

Processing speed on r4.2xlarge (working on 7 cpu)

model	insType	time_mins
Mlmuk1a	r4.2xlarge	0.141056816
Mbglmuk1a	r4.2xlarge	1.147005661
Mgamuk1a	r4.2xlarge	8.452761634
Mgbmuk1a	r4.2xlarge	2.072490887
Msvmuk1a	r4.2xlarge	8.794263363
Mknnuk1a	r4.2xlarge	1.755700115
Mglmuk1a	r4.2xlarge	0.052517164
Mlmuk1a.full	r4.2xlarge	0.059232525

bogsnork commented 6 years ago

rf run failed on c4.8xlarge instance

model	insType	time_mins
Mlmuk1a	c4.8xlarge	0.1640143 mins
Mrfuk1a - failed	c4.8xlarge	248.4563628 mins

bogsnork commented 6 years ago

next step

[x] bring models together and evaluate (using 03 script and ff), probably won't need a huge instance.
[x] write script for ensembling
[x] produce output raster

then move to interpolation

bogsnork commented 6 years ago

Evaluate all models

Random Forest model performs best all all metrics, followed by General Additive Model and Gradient Boosted Model

model	RMSE	Rsquared	MAE
lm	72.32062	0.2199118	51.79360
rf	49.53429	0.6340381	30.54394
glm	72.32062	0.2199118	51.79360
glmb	75.96879	0.1407866	55.08493
gam	62.42566	0.4189245	44.02222
gbm	59.96117	0.4679294	42.48843
svm	75.50815	0.1798493	48.77628

Call: summary.diff.resamples(object = Diffs)

p-value adjustment: bonferroni Upper diagonal: estimates of the difference Lower diagonal: p-value for H0: difference = 0

bogsnork commented 6 years ago

RF predictions

pd_rf_moduk1a.tif ... doesn't look very plausible

bogsnork commented 6 years ago

Launched new EC2 instance enPeatDepthModel 5 r4.4xlarge with additional storage (100Gb) as prev instance too small to store output tifs and predicting above took over 1 hour.