Ensemble forecasting with future climates issue

[CatherineBuckland]

Hi biomod2 team,

I have been having a play with the biomod2 package - in particular we are interested in building an ensemble model of species distribution for one particular species, and then using future climate datasets, projecting future species distributions based on the ensemble model built on current conditions. When I build ensemble models (currently building weighted means and CV for 5 different eval metrics) for current conditions, the ten plots of species distribution all look sensible. However, when I then feed in new future climate datasets, complete new projections and then ensemble forecast - my new future species distribution plots look awful. The weighted mean maps are basically blank and the CV maps show high / medium levels of uncertainty. I've been having a play, and if I didn't complete any ensemble modelling - i.e. if I just ran one Random Forest model on current conditions - then the current and all future climate maps based on this one algorithm all produce sensible looking results. Obviously I would rather use the ensemble modelling function to build-in further robustness to the results and also highlight the uncertainty in the outputs, but I am a bit stuck on what is happening during the ensemble forecasting stage with the future climates that is causing such awful results (especially since when this same future climates data when used on one single algorithm produces sensible results).

Any advice / help would be great!

Thanks, Catherine

[DamienGeorges]

Hi,

Did you try to investigate individual projections in future climatic conditions? That is to say do the projection of each individual model via BIOMOD_projection and have a look to individual maps to check if you can find a systematic bias according to model chosen? In the meantime, you can double check that your curent and future environmental conditions are somehow overlap (no transformation of the future conditions).

Did you use the rescall.model option at modelling stage?

Best, Damien

[CatherineBuckland]

Hi Damien,

Many thanks for getting back to me. Since I posted my query I started looking into the individual projections for future climatic conditions – and realised that the problem must be in this step (rather than ensemble modelling) because each of the individual projections looks almost completely blank (a few had very small distributions but very low certainty). The model evaluation scores are all really high, often > 0.95 (across range of evaluation metrics). I have tried all sorts of different things to see what can improve the future climate projections: altering number of environmental variables used, number of PA repeats, data split, etc.

Following your email I have switched off the rescal.model option. I realised I think I just selected this as TRUE to begin with – but I’m not entirely sure what it does? Since switching it off, my future projections are producing results – albeit they still have low certainty for any distributions, and the CV ensemble model is blank… which doesn’t make any sense since the individual projections show a lot of variability. I have also switched off the do.full.models option – again, could you explain what this does?

You mentioned that perhaps I should look at the variation between the current and future environmental conditions, do you know how can I double check that the current and future environmental conditions overlap?

I really want to understand what it is that is causing the outputs to look like this because once I fully understand what the model is doing, I will be able to have confidence in the results 😊

It would be great to hear if you have any suggestions / advice, and let me know if it would just be easier for me to share the script that I have got so far.

Best

Catherine

From: Damien Georges notifications@github.com Sent: 20 February 2020 15:49 To: biomodhub/biomod2 biomod2@noreply.github.com Cc: Catherine Buckland catherine.buckland@ouce.ox.ac.uk; Author author@noreply.github.com Subject: Re: [biomodhub/biomod2] Ensemble forecasting with future climates issue (#11)

Hi,

Did you try to investigate individual projections in future climatic conditions? That is to say do the projection of each individual model via BIOMOD_projection and have a look to individual maps to check if you can find a systematic bias according to model chosen? In the meantime, you can double check that your curent and future environmental conditions are somehow overlap (no transformation of the future conditions).

Did you use the rescall.model option at modelling stage?

Best, Damien

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/biomodhub/biomod2/issues/11?email_source=notifications&email_token=ALHXM6N3K3RLRFP6A3GGQXDRD2Q45A5CNFSM4KX4SAQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMO244I#issuecomment-589147761, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALHXM6IADGP2LAGHPRBC233RD2Q45ANCNFSM4KX4SAQA.

[DamienGeorges]

Hi,

do.full.models option compute a model with 100% of the data for calibration and evaluate the model on the same set of data (No CV). Then model evaluation are overoptimistic since you evaluate a model with data it has been trained with but might be useful when you have very few occurence point.

I often advise people not to userescall.model option.

Hope that helps, Damien

[Roicarba]

Hi, I have the same issue and I don`t understand where I can change the rescall.model. When I compute the ensemble?