biomodhub / biomod2

BIOMOD is a computer platform for ensemble forecasting of species distributions, enabling the treatment of a range of methodological uncertainties in models and the examination of species-environment relationships.
87 stars 22 forks source link

Unexpected error in GAM algorithm of BIOMOD.modeling #293

Closed LorenzoBernicchi closed 1 year ago

LorenzoBernicchi commented 1 year ago

Hello everyone, I am modeling a species distribution as I have done hundreds of times before.

I am using six different algorithms, namely GLM, GBM, GAM, RF, MAXENT and MAXNET. I did not encounter any problems, except for the GAM. This algorithm takes a very long time to finish a single run (sometimes more than one hour) and in some of them I received this error: Error in gam.fit3(x = X, y = y, sp = L %*% lsp3 + lsp0, Eb = Eb, UrS = UrS, : inner loop 3; can't correct step size. I am using the default modeling options for every algorithms, and I already used the same modeling procedure with the same data but this is the first time I am facing this error.

Hope you can help me, let me know if you need additional information. Thanks in advance, best regards.

Lorenzo Bernicchi

rpatin commented 1 year ago

Hello Lorenzo, Thank you for reporting :pray: Unfortunately we are no gam experts. I found some (old) post related to that error:

That error message and the warning indicate that the algorithm (iteratively reweighted least squares or IRLS) for fitting the parameters cannot converge, which can be because the model is over-specified. As you see, when you remove of the terms you do get parameter estimates and I would guess that, even then, the model will be over-specified.

So maybe this is an issue raising in some model that are overparameterized (which can depends on the randomness of the datasplit if you use cross-validation).

From there, they advise on using REML method, which might help in solving (or understanding your issue if this still fail). You can change the gam method as follows:

myBiomodOptions <- BIOMOD_ModelingOptions(GAM = list("method" = "REML"))

If that does not help, you might have to ask the question to people more used to gam.

Best regards, Rémi

LorenzoBernicchi commented 1 year ago

Dear @rpatin, thank you very much for your answer!

I tried changing the method in the gam algorithm, but nothing seems to have changed. This time, however, the errore message looks a bit different, here you can read it:

Error in gam.fit3(x = X, y = y, sp = L %*% lsp3 + lsp0, Eb = Eb, UrS = UrS, : inner loop 3; can't correct step size In addition: Warning messages: 1: In bm_RunModelsLoop(bm.format = bm.format, weights = weights, calib.lines = calib.lines,: Parallelisation with foreach is not available for Windows. Sorry. 2: executing %dopar% sequentially: no parallel backend registered Error in h(simpleError(msg, call)) : errore durante la valutazione dell'argomento 'object' nella selezione di un metodo per la funzione 'predict': object 'model.bm' not found

Would you suggest me to ask this question to someone more used to gam as you said? May I ask you where I can post this question? Maybe on stackexchange?

Thank you in advance, I wish you a good day!

rpatin commented 1 year ago

Dear Lorenzo, Thank you for the update. You may also try using package gam instead of mgcv:

myBiomodOptions <- BIOMOD_ModelingOptions(GAM = list("algo" = "GAM_gam")) # edit fixed mistake in "GAM_gam"

If that still does not work, you can indeed try to look for help elsewhere. I would have advised stackoverflow rather than stackexchange as it is more a technical issue rather than a statistical question (however I am no expert of those websites).

If you finally get an answer or solution, it would be great if you could update the issue with it.

Best regards, Rémi

LorenzoBernicchi commented 1 year ago

Dear @rpatin, So you would suggest me to specify the BIOMOD_ModelingOptions like this: myBIOMODoption <- BIOMOD_ModelingOptions( GAM = list(algo = "GAM_mgcv", method = "REML"))? Should I discard the method part, and moreover should I use the GAM_gam or GAM_mgcv? I am afraid I misunderstood you.

Thank you very much!

rpatin commented 1 year ago

Dear Lorenzo, Just as follows:

myBiomodOptions <- BIOMOD_ModelingOptions(GAM = list("algo" = "GAM_gam"))

Although it would not matter if you kept method = "REML" it would just be ignored. Sorry for the typo in "GAM_gam" it should be clearer now !

Best, Rémi

LorenzoBernicchi commented 1 year ago

Thank you very much, I will try like this now and if it will not work I will ask on stackoverflow.

Thank you again, I wish you a good day!

rpatin commented 1 year ago

Great, feel free to post here the link to your question. I can have a look if there are points you are not sure how to manage through biomod2. Thanks, have a good day as well!

LorenzoBernicchi commented 1 year ago

It's still me, I have a little update:

I tried to use the GAM_gam algorithm, but on the console appears only this, and after few seconds (like 2-3) the next algorithm start to process and it seems like the gam isn't working:

I am using the GAM_gam as algorithm and, after just 2-3 seconds, I get this in return:

-=-=-=--=-=-=- Capr.Carso_PA1_RUN2_GAM 

Model=GAM
     GAM_gam algorithm chosen

Moreover, after processing the last algorithm of the last run, I get this error message:

Error in { : 
  task 3 failed - "the namespace ‘mgcv’ is imported from ‘vegan’ thus cannot be downloaded"
In addition: Warning messages:
1: In bm_RunModelsLoop(bm.format = bm.format, weights = weights, calib.lines = calib.lines,  :
  Parallelisation with `foreach` is not available for Windows. Sorry.
2: executing %dopar% sequentially: no parallel backend registered 

I take a look on my folders and all the single models were correctly created except for the GAM.

However, I did have on my R environment the vegan package and I tried also to upload it, but nothing changed.

rpatin commented 1 year ago

Package gam and mgcv are quite in conflict and it is best that they are not loaded together. The error message is slightly misleading, what you should do is unloading package vegan:

detach("package:vegan", unload=TRUE)

Hopefully this should work. Best, Rémi

LorenzoBernicchi commented 1 year ago

Dear @rpatin ,

I tried again and this time I get something a bit different.

When the gam started working, I could read this in my console:

-=-=-=--=-=-=- Capr.Carso_PA1_RUN10_GAM 

Model=GAM
     GAM_gam algorithm chosen
*** single value predicted
   ! Note :  Capr.Carso_PA1_RUN10_GAM failed!

Do you know why this could happen?

rpatin commented 1 year ago

Dear Lorenzo, Perhaps, we are starting to see a way out then. Is it happening for all the cross-validation fold ? Or just some of them ? It would be interesting to see which fold the GAM is failing with and look at the data within that fold.

you can do something as follows to start exploring this idea:

plot(myBiomodData, calib.lines = myBiomodCV, 
     run = "RUN10", PA = "PA1") # use plot.type = "raster" if you have a huge dataset, this can be nicer
LorenzoBernicchi commented 1 year ago

It happened for ALL the cross-validation (10 times since I set CV.nb.rep = 10). I am trying to look at what you asked me, but may I ask you what should be the myBiomodCV object?

rpatin commented 1 year ago

Sorry, myBiomodCV was the output of either:

If that's common to all your fold, this may be easy to reproduce. I could try to have a look if you can send me the dataset (remi.patin@univ-grenoble-alpes.fr).

Does your other algorithm are OK ? they have good performance and are not overfitting for instance ?

LorenzoBernicchi commented 1 year ago

I am modeling a species in a sub-region of Italy (quite small actually) using only 35 presence points of my species.

This is the result of the plot you asked me before, if this is not what you were looking for just ask me:

Rplot

However, my other algorithm do not seem ok. The RF is still overfitting (with all the TSS and AUC values that are 1, I don't know if you remember an old issue I posted last month) and maybe also the GBM doesn't work as it should. I attached here a table with the scores of all models.

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

algo | metric.eval | cutoff | sensitivity | specificity | calibration | validation | evaluation -- | -- | -- | -- | -- | -- | -- | -- GLM | TSS | 478 | 83,333 | 58,886 | 0,424 | 0,413 | NA GLM | ROC | 478,5 | 83,333 | 59,043 | 0,665 | 0,649 | NA GBM | TSS | 514 | 100 | 98,414 | 0,984 | 0,163 | NA GBM | ROC | 520,5 | 100 | 98,514 | 0,996 | 0,885 | NA RF | TSS | 143 | 100 | 100 | 1 | 0 | NA RF | ROC | 143 | 100 | 100 | 1 | 0,679 | NA MAXENT | TSS | 487 | 79,167 | 90,771 | 0,7 | 0,354 | NA MAXENT | ROC | 488,5 | 79,167 | 90,914 | 0,902 | 0,817 | NA MAXNET | TSS | 488 | 75 | 90,471 | 0,655 | 0,349 | NA MAXNET | ROC | 491,5 | 75 | 90,571 | 0,889 | 0,792 | NA GLM | TSS | 440 | 91,667 | 57,671 | 0,495 | 0,302 | NA GLM | ROC | 444,5 | 91,667 | 58,671 | 0,738 | 0,643 | NA GBM | TSS | 543 | 100 | 98,671 | 0,987 | 0,166 | NA GBM | ROC | 548 | 100 | 98,714 | 0,995 | 0,693 | NA RF | TSS | 185 | 100 | 100 | 1 | 0 | NA RF | ROC | 186 | 100 | 100 | 1 | 0,657 | NA MAXENT | TSS | 216 | 95,833 | 76,986 | 0,729 | 0,039 | NA MAXENT | ROC | 217,5 | 95,833 | 77,243 | 0,931 | 0,682 | NA MAXNET | TSS | 213 | 100 | 74,3 | 0,745 | 0,201 | NA MAXNET | ROC | 222,5 | 100 | 76,171 | 0,93 | 0,694 | NA GLM | TSS | 407 | 87,5 | 65,886 | 0,535 | 0,389 | NA GLM | ROC | 412,5 | 87,5 | 66,386 | 0,839 | 0,71 | NA GBM | TSS | 544 | 100 | 98,814 | 0,988 | -0,012 | NA GBM | ROC | 549,5 | 100 | 98,857 | 0,996 | 0,741 | NA RF | TSS | 181 | 100 | 100 | 1 | 0 | NA RF | ROC | 181 | 100 | 100 | 1 | 0,648 | NA MAXENT | TSS | 226 | 95,833 | 77,329 | 0,732 | 0,319 | NA MAXENT | ROC | 233,5 | 95,833 | 78,2 | 0,932 | 0,751 | NA MAXNET | TSS | 411 | 83,333 | 88,186 | 0,716 | 0,153 | NA MAXNET | ROC | 412,5 | 83,333 | 88,386 | 0,911 | 0,741 | NA

rpatin commented 1 year ago

Thank you for the update.

The plot was good, thank you ! It may be easier to read with plot.type = "raster" however.

Considering everything I think the issue is more generic than the GAM error. How many pseudo-absences did you consider ? With such imbalance and few presence points (35) I do not think it is surprising that the GAM is failing to fit. You clearly can see overfit for RF, but also for other models (GBM for instance, but also most of the other one to be honnest). The difference is more important for TSS than ROC, as TSS is threshold dependent, and the threshold calculated on calibration data seems not appropriate for the validation dataset. Although we should also consider that your validation dataset have very few presences so it might be hard to draw conclusion from the TSS. You can check the content of your fold using:

summary(myBiomodData, calib.lines = myBiomodCV)

Concerning the overfit situation:

Best, Rémi

LorenzoBernicchi commented 1 year ago

Dear @rpatin,

Today I am doing a bit different modelization: I am building this model over Europe considering the 19 WolrdClim variables and using more than 3'500 points of presence. However, I am still facing the same problems with the randomForest as well as pretty bad evaluation scores for the other algorithms. I really hope you can help me out.

Hereafter, I show you my data, the modeling steps and the results I get so far.

Capreolus_BIOMOD_data <- BIOMOD_FormatingData(
  resp.name = "Capreolus Capreolus",
  resp.var = Capreolus_points,
  expl.var = Bioclim_Capreolus,
  PA.nb.rep = 1,
  PA.nb.absences = 20000,
  PA.strategy = 'random'
)
Capreolus_BIOMOD_data

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= BIOMOD.formated.data -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

dir.name =  .

sp.name =  Capreolus.Capreolus

     3064 presences,  0 true absences and  20000 undefined points in dataset

     7 explanatory variables

     bio_2            bio_7            bio_10          bio_11           bio_12           bio_14      
 Min.   : 1.851   Min.   : 4.496   Min.   :  6.0   Min.   :  25.0   Min.   : 3.220   Min.   : 273.9  
 1st Qu.:14.422   1st Qu.:24.001   1st Qu.:169.0   1st Qu.: 102.0   1st Qu.: 7.824   1st Qu.: 665.7  
 Median :16.426   Median :30.488   Median :203.0   Median : 119.0   Median : 8.464   Median : 824.1  
 Mean   :16.585   Mean   :30.349   Mean   :197.7   Mean   : 154.8   Mean   : 8.591   Mean   : 837.1  
 3rd Qu.:18.714   3rd Qu.:34.727   3rd Qu.:226.0   3rd Qu.: 173.0   3rd Qu.: 9.355   3rd Qu.:1002.4  
 Max.   :27.113   Max.   :99.578   Max.   :659.0   Max.   :1148.0   Max.   :15.065   Max.   :1414.0  
     bio_18      
 Min.   :-9.449  
 1st Qu.: 9.669  
 Median :14.036  
 Mean   :12.933  
 3rd Qu.:16.842  
 Max.   :23.394  

 1 Pseudo Absences dataset available ( PA1 ) with  20000 (PA1) pseudo absences

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

myBIOMODoption <- BIOMOD_ModelingOptions()

Capreolus_single_models <- BIOMOD_Modeling(
  bm.format = Capreolus_BIOMOD_data,
  modeling.id = "Single_models",
  models = c("GLM", "GBM", "GAM", "RF", "MAXENT", "MAXNET"),
  bm.options = myBIOMODoption,
  CV.strategy = "random",
  CV.nb.rep = 10,
  CV.perc = 0.7,
  CV.do.full.models = F,
  metric.eval = c("ROC", "TSS"),
  var.import = 1,
  nb.cpu = 4,
  seed.val = 42,
  do.progress = T)

scores <- get_evaluations(Capreolus_single_models)
scores

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

  | full.name | PA | run | algo | metric.eval | cutoff | sensitivity | specificity | calibration | validation | evaluation -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- 1 | Capreolus.Capreolus_PA1_RUN1_GLM | PA1 | RUN1 | GLM | ROC | 496,5 | 81,725 | 55,757 | 0,744 | 0,749 | NA 2 | Capreolus.Capreolus_PA1_RUN1_GLM | PA1 | RUN1 | GLM | TSS | 498 | 81,538 | 55,85 | 0,373 | 0,386 | NA 3 | Capreolus.Capreolus_PA1_RUN1_GBM | PA1 | RUN1 | GBM | ROC | 550,5 | 77,855 | 63,1 | 0,76 | 0,756 | NA 4 | Capreolus.Capreolus_PA1_RUN1_GBM | PA1 | RUN1 | GBM | TSS | 550 | 78,182 | 62,757 | 0,41 | 0,403 | NA 5 | Capreolus.Capreolus_PA1_RUN1_GAM | PA1 | RUN1 | GAM | ROC | 463,5 | 84,709 | 54,493 | 0,752 | 0,754 | NA 6 | Capreolus.Capreolus_PA1_RUN1_GAM | PA1 | RUN1 | GAM | TSS | 459 | 85,128 | 53,936 | 0,391 | 0,385 | NA 7 | Capreolus.Capreolus_PA1_RUN1_RF | PA1 | RUN1 | RF | ROC | 370 | 99,907 | 99,943 | 1 | 0,728 | NA 8 | Capreolus.Capreolus_PA1_RUN1_RF | PA1 | RUN1 | RF | TSS | 330 | 100 | 99,836 | 0,998 | 0,048 | NA 9 | Capreolus.Capreolus_PA1_RUN1_MAXENT | PA1 | RUN1 | MAXENT | ROC | 434,5 | 76,643 | 63,029 | 0,762 | 0,755 | NA 10 | Capreolus.Capreolus_PA1_RUN1_MAXENT | PA1 | RUN1 | MAXENT | TSS | 445 | 75,058 | 64,514 | 0,396 | 0,402 | NA 11 | Capreolus.Capreolus_PA1_RUN1_MAXNET | PA1 | RUN1 | MAXNET | ROC | 406,5 | 81,818 | 56,421 | 0,75 | 0,753 | NA 12 | Capreolus.Capreolus_PA1_RUN1_MAXNET | PA1 | RUN1 | MAXNET | TSS | 408 | 81,632 | 56,593 | 0,382 | 0,394 | NA 13 | Capreolus.Capreolus_PA1_RUN2_GLM | PA1 | RUN2 | GLM | ROC | 515,5 | 79,441 | 58,393 | 0,744 | 0,745 | NA 14 | Capreolus.Capreolus_PA1_RUN2_GLM | PA1 | RUN2 | GLM | TSS | 514 | 79,487 | 58,171 | 0,377 | 0,376 | NA 15 | Capreolus.Capreolus_PA1_RUN2_GBM | PA1 | RUN2 | GBM | ROC | 542,5 | 78,928 | 61,886 | 0,761 | 0,751 | NA 16 | Capreolus.Capreolus_PA1_RUN2_GBM | PA1 | RUN2 | GBM | TSS | 561 | 75,711 | 65,036 | 0,407 | 0,388 | NA 17 | Capreolus.Capreolus_PA1_RUN2_GAM | PA1 | RUN2 | GAM | ROC | 478,5 | 82,517 | 57,4 | 0,755 | 0,747 | NA 18 | Capreolus.Capreolus_PA1_RUN2_GAM | PA1 | RUN2 | GAM | TSS | 476 | 82,611 | 57,036 | 0,397 | 0,385 | NA 19 | Capreolus.Capreolus_PA1_RUN2_RF | PA1 | RUN2 | RF | ROC | 353 | 100 | 99,929 | 1 | 0,723 | NA 20 | Capreolus.Capreolus_PA1_RUN2_RF | PA1 | RUN2 | RF | TSS | 347 | 100 | 99,907 | 0,999 | 0,048 | NA 21 | Capreolus.Capreolus_PA1_RUN2_MAXENT | PA1 | RUN2 | MAXENT | ROC | 422,5 | 78,322 | 62,45 | 0,766 | 0,749 | NA 22 | Capreolus.Capreolus_PA1_RUN2_MAXENT | PA1 | RUN2 | MAXENT | TSS | 422 | 78,368 | 62,264 | 0,408 | 0,379 | NA 23 | Capreolus.Capreolus_PA1_RUN2_MAXNET | PA1 | RUN2 | MAXNET | ROC | 417,5 | 80,186 | 58,85 | 0,753 | 0,751 | NA 24 | Capreolus.Capreolus_PA1_RUN2_MAXNET | PA1 | RUN2 | MAXNET | TSS | 418 | 80,186 | 58,85 | 0,389 | 0,386 | NA 25 | Capreolus.Capreolus_PA1_RUN3_GLM | PA1 | RUN3 | GLM | ROC | 518,5 | 78,974 | 59,829 | 0,751 | 0,732 | NA 26 | Capreolus.Capreolus_PA1_RUN3_GLM | PA1 | RUN3 | GLM | TSS | 513 | 79,534 | 59,2 | 0,387 | 0,354 | NA 27 | Capreolus.Capreolus_PA1_RUN3_GBM | PA1 | RUN3 | GBM | ROC | 535,5 | 80,653 | 61,921 | 0,766 | 0,736 | NA 28 | Capreolus.Capreolus_PA1_RUN3_GBM | PA1 | RUN3 | GBM | TSS | 535 | 80,746 | 61,75 | 0,426 | 0,365 | NA 29 | Capreolus.Capreolus_PA1_RUN3_GAM | PA1 | RUN3 | GAM | ROC | 489,5 | 82,331 | 58,686 | 0,76 | 0,733 | NA 30 | Capreolus.Capreolus_PA1_RUN3_GAM | PA1 | RUN3 | GAM | TSS | 488 | 82,517 | 58,493 | 0,41 | 0,368 | NA 31 | Capreolus.Capreolus_PA1_RUN3_RF | PA1 | RUN3 | RF | ROC | 368 | 100 | 99,957 | 1 | 0,716 | NA 32 | Capreolus.Capreolus_PA1_RUN3_RF | PA1 | RUN3 | RF | TSS | 365 | 100 | 99,957 | 1 | 0,047 | NA 33 | Capreolus.Capreolus_PA1_RUN3_MAXENT | PA1 | RUN3 | MAXENT | ROC | 370,5 | 83,497 | 58,414 | 0,772 | 0,735 | NA 34 | Capreolus.Capreolus_PA1_RUN3_MAXENT | PA1 | RUN3 | MAXENT | TSS | 370 | 83,543 | 58,329 | 0,419 | 0,356 | NA 35 | Capreolus.Capreolus_PA1_RUN3_MAXNET | PA1 | RUN3 | MAXNET | ROC | 405,5 | 81,352 | 58,514 | 0,758 | 0,737 | NA 36 | Capreolus.Capreolus_PA1_RUN3_MAXNET | PA1 | RUN3 | MAXNET | TSS | 406 | 81,352 | 58,514 | 0,398 | 0,357 | NA 37 | Capreolus.Capreolus_PA1_RUN4_GLM | PA1 | RUN4 | GLM | ROC | 525,5 | 78,555 | 59,757 | 0,745 | 0,745 | NA 38 | Capreolus.Capreolus_PA1_RUN4_GLM | PA1 | RUN4 | GLM | TSS | 526 | 78,555 | 59,757 | 0,383 | 0,367 | NA 39 | Capreolus.Capreolus_PA1_RUN4_GBM | PA1 | RUN4 | GBM | ROC | 555,5 | 76,643 | 65,05 | 0,76 | 0,752 | NA 40 | Capreolus.Capreolus_PA1_RUN4_GBM | PA1 | RUN4 | GBM | TSS | 555 | 76,783 | 64,779 | 0,417 | 0,389 | NA 41 | Capreolus.Capreolus_PA1_RUN4_GAM | PA1 | RUN4 | GAM | ROC | 523,5 | 78,089 | 61,993 | 0,755 | 0,746 | NA 42 | Capreolus.Capreolus_PA1_RUN4_GAM | PA1 | RUN4 | GAM | TSS | 490 | 81,492 | 58,35 | 0,399 | 0,38 | NA 43 | Capreolus.Capreolus_PA1_RUN4_RF | PA1 | RUN4 | RF | ROC | 371 | 100 | 99,964 | 1 | 0,737 | NA 44 | Capreolus.Capreolus_PA1_RUN4_RF | PA1 | RUN4 | RF | TSS | 367 | 100 | 99,957 | 1 | 0,047 | NA 45 | Capreolus.Capreolus_PA1_RUN4_MAXENT | PA1 | RUN4 | MAXENT | ROC | 424,5 | 78,182 | 62,671 | 0,765 | 0,75 | NA 46 | Capreolus.Capreolus_PA1_RUN4_MAXENT | PA1 | RUN4 | MAXENT | TSS | 405 | 80,233 | 60,293 | 0,407 | 0,379 | NA 47 | Capreolus.Capreolus_PA1_RUN4_MAXNET | PA1 | RUN4 | MAXNET | ROC | 417,5 | 80,326 | 59,014 | 0,754 | 0,749 | NA 48 | Capreolus.Capreolus_PA1_RUN4_MAXNET | PA1 | RUN4 | MAXNET | TSS | 415 | 80,513 | 58,586 | 0,392 | 0,377 | NA 49 | Capreolus.Capreolus_PA1_RUN5_GLM | PA1 | RUN5 | GLM | ROC | 512,5 | 79,86 | 58,136 | 0,745 | 0,745 | NA 50 | Capreolus.Capreolus_PA1_RUN5_GLM | PA1 | RUN5 | GLM | TSS | 501 | 81,119 | 56,729 | 0,38 | 0,377 | NA 51 | Capreolus.Capreolus_PA1_RUN5_GBM | PA1 | RUN5 | GBM | ROC | 540,5 | 79,86 | 61,386 | 0,76 | 0,751 | NA 52 | Capreolus.Capreolus_PA1_RUN5_GBM | PA1 | RUN5 | GBM | TSS | 539 | 80,093 | 60,979 | 0,411 | 0,399 | NA 53 | Capreolus.Capreolus_PA1_RUN5_GAM | PA1 | RUN5 | GAM | ROC | 471,5 | 83,73 | 56,129 | 0,752 | 0,752 | NA 54 | Capreolus.Capreolus_PA1_RUN5_GAM | PA1 | RUN5 | GAM | TSS | 475 | 83,31 | 56,471 | 0,398 | 0,389 | NA 55 | Capreolus.Capreolus_PA1_RUN5_RF | PA1 | RUN5 | RF | ROC | 361 | 100 | 99,95 | 1 | 0,734 | NA 56 | Capreolus.Capreolus_PA1_RUN5_RF | PA1 | RUN5 | RF | TSS | 361 | 100 | 99,95 | 1 | 0,049 | NA 57 | Capreolus.Capreolus_PA1_RUN5_MAXENT | PA1 | RUN5 | MAXENT | ROC | 388,5 | 82,751 | 57,879 | 0,762 | 0,754 | NA 58 | Capreolus.Capreolus_PA1_RUN5_MAXENT | PA1 | RUN5 | MAXENT | TSS | 385 | 82,844 | 57,521 | 0,404 | 0,376 | NA 59 | Capreolus.Capreolus_PA1_RUN5_MAXNET | PA1 | RUN5 | MAXNET | ROC | 398,5 | 82,751 | 55,75 | 0,75 | 0,751 | NA 60 | Capreolus.Capreolus_PA1_RUN5_MAXNET | PA1 | RUN5 | MAXNET | TSS | 396 | 83,17 | 55,271 | 0,385 | 0,384 | NA 61 | Capreolus.Capreolus_PA1_RUN6_GLM | PA1 | RUN6 | GLM | ROC | 529,5 | 78,275 | 60,379 | 0,748 | 0,736 | NA 62 | Capreolus.Capreolus_PA1_RUN6_GLM | PA1 | RUN6 | GLM | TSS | 530 | 78,275 | 60,379 | 0,386 | 0,343 | NA 63 | Capreolus.Capreolus_PA1_RUN6_GBM | PA1 | RUN6 | GBM | ROC | 525,5 | 82,051 | 59,643 | 0,762 | 0,744 | NA 64 | Capreolus.Capreolus_PA1_RUN6_GBM | PA1 | RUN6 | GBM | TSS | 524 | 82,424 | 59,257 | 0,416 | 0,376 | NA 65 | Capreolus.Capreolus_PA1_RUN6_GAM | PA1 | RUN6 | GAM | ROC | 483,5 | 82,564 | 57,771 | 0,755 | 0,746 | NA 66 | Capreolus.Capreolus_PA1_RUN6_GAM | PA1 | RUN6 | GAM | TSS | 489 | 81,958 | 58,307 | 0,403 | 0,367 | NA 67 | Capreolus.Capreolus_PA1_RUN6_RF | PA1 | RUN6 | RF | ROC | 359 | 100 | 99,964 | 1 | 0,735 | NA 68 | Capreolus.Capreolus_PA1_RUN6_RF | PA1 | RUN6 | RF | TSS | 353 | 100 | 99,957 | 1 | 0,06 | NA 69 | Capreolus.Capreolus_PA1_RUN6_MAXENT | PA1 | RUN6 | MAXENT | ROC | 407,5 | 79,814 | 61,136 | 0,765 | 0,744 | NA 70 | Capreolus.Capreolus_PA1_RUN6_MAXENT | PA1 | RUN6 | MAXENT | TSS | 409 | 79,627 | 61,286 | 0,408 | 0,368 | NA 71 | Capreolus.Capreolus_PA1_RUN6_MAXNET | PA1 | RUN6 | MAXNET | ROC | 431,5 | 78,182 | 61,236 | 0,754 | 0,742 | NA 72 | Capreolus.Capreolus_PA1_RUN6_MAXNET | PA1 | RUN6 | MAXNET | TSS | 436 | 77,576 | 61,736 | 0,394 | 0,365 | NA 73 | Capreolus.Capreolus_PA1_RUN7_GLM | PA1 | RUN7 | GLM | ROC | 507,5 | 80,979 | 57,657 | 0,751 | 0,733 | NA 74 | Capreolus.Capreolus_PA1_RUN7_GLM | PA1 | RUN7 | GLM | TSS | 520 | 79,674 | 58,843 | 0,386 | 0,36 | NA 75 | Capreolus.Capreolus_PA1_RUN7_GBM | PA1 | RUN7 | GBM | ROC | 530,5 | 80,886 | 60,607 | 0,764 | 0,741 | NA 76 | Capreolus.Capreolus_PA1_RUN7_GBM | PA1 | RUN7 | GBM | TSS | 528 | 81,445 | 60 | 0,415 | 0,386 | NA 77 | Capreolus.Capreolus_PA1_RUN7_GAM | PA1 | RUN7 | GAM | ROC | 523,5 | 78,741 | 62,35 | 0,759 | 0,737 | NA 78 | Capreolus.Capreolus_PA1_RUN7_GAM | PA1 | RUN7 | GAM | TSS | 528 | 77,855 | 62,779 | 0,407 | 0,357 | NA 79 | Capreolus.Capreolus_PA1_RUN7_RF | PA1 | RUN7 | RF | ROC | 363 | 99,907 | 99,929 | 1 | 0,717 | NA 80 | Capreolus.Capreolus_PA1_RUN7_RF | PA1 | RUN7 | RF | TSS | 364 | 99,907 | 99,929 | 0,998 | 0,041 | NA 81 | Capreolus.Capreolus_PA1_RUN7_MAXENT | PA1 | RUN7 | MAXENT | ROC | 420,5 | 77,762 | 63,343 | 0,769 | 0,737 | NA 82 | Capreolus.Capreolus_PA1_RUN7_MAXENT | PA1 | RUN7 | MAXENT | TSS | 422 | 77,622 | 63,436 | 0,411 | 0,35 | NA 83 | Capreolus.Capreolus_PA1_RUN7_MAXNET | PA1 | RUN7 | MAXNET | ROC | 415,5 | 80 | 59,414 | 0,756 | 0,738 | NA 84 | Capreolus.Capreolus_PA1_RUN7_MAXNET | PA1 | RUN7 | MAXNET | TSS | 418 | 79,72 | 59,65 | 0,393 | 0,366 | NA 85 | Capreolus.Capreolus_PA1_RUN8_GLM | PA1 | RUN8 | GLM | ROC | 521,5 | 78,974 | 59,807 | 0,751 | 0,73 | NA 86 | Capreolus.Capreolus_PA1_RUN8_GLM | PA1 | RUN8 | GLM | TSS | 524 | 78,741 | 59,971 | 0,388 | 0,348 | NA 87 | Capreolus.Capreolus_PA1_RUN8_GBM | PA1 | RUN8 | GBM | ROC | 542,5 | 78,881 | 63,536 | 0,767 | 0,736 | NA 88 | Capreolus.Capreolus_PA1_RUN8_GBM | PA1 | RUN8 | GBM | TSS | 543 | 78,881 | 63,536 | 0,424 | 0,369 | NA 89 | Capreolus.Capreolus_PA1_RUN8_GAM | PA1 | RUN8 | GAM | ROC | 514,5 | 79,254 | 61,407 | 0,759 | 0,739 | NA 90 | Capreolus.Capreolus_PA1_RUN8_GAM | PA1 | RUN8 | GAM | TSS | 501 | 80,653 | 59,836 | 0,405 | 0,364 | NA 91 | Capreolus.Capreolus_PA1_RUN8_RF | PA1 | RUN8 | RF | ROC | 360 | 100 | 99,921 | 1 | 0,725 | NA 92 | Capreolus.Capreolus_PA1_RUN8_RF | PA1 | RUN8 | RF | TSS | 360 | 100 | 99,921 | 0,999 | 0,046 | NA 93 | Capreolus.Capreolus_PA1_RUN8_MAXENT | PA1 | RUN8 | MAXENT | ROC | 427,5 | 76,41 | 64,693 | 0,769 | 0,736 | NA 94 | Capreolus.Capreolus_PA1_RUN8_MAXENT | PA1 | RUN8 | MAXENT | TSS | 426 | 76,643 | 64,386 | 0,411 | 0,361 | NA 95 | Capreolus.Capreolus_PA1_RUN8_MAXNET | PA1 | RUN8 | MAXNET | ROC | 434,5 | 77,249 | 62,421 | 0,758 | 0,733 | NA 96 | Capreolus.Capreolus_PA1_RUN8_MAXNET | PA1 | RUN8 | MAXNET | TSS | 432 | 77,529 | 61,986 | 0,396 | 0,353 | NA 97 | Capreolus.Capreolus_PA1_RUN9_GLM | PA1 | RUN9 | GLM | ROC | 515,5 | 79,907 | 57,914 | 0,747 | 0,742 | NA 98 | Capreolus.Capreolus_PA1_RUN9_GLM | PA1 | RUN9 | GLM | TSS | 515 | 79,953 | 57,743 | 0,378 | 0,378 | NA 99 | Capreolus.Capreolus_PA1_RUN9_GBM | PA1 | RUN9 | GBM | ROC | 555,5 | 76,503 | 64,436 | 0,764 | 0,744 | NA 100 | Capreolus.Capreolus_PA1_RUN9_GBM | PA1 | RUN9 | GBM | TSS | 565 | 74,545 | 66,229 | 0,409 | 0,383 | NA 101 | Capreolus.Capreolus_PA1_RUN9_GAM | PA1 | RUN9 | GAM | ROC | 509,5 | 80,233 | 60 | 0,757 | 0,741 | NA 102 | Capreolus.Capreolus_PA1_RUN9_GAM | PA1 | RUN9 | GAM | TSS | 503 | 80,746 | 59,2 | 0,4 | 0,375 | NA 103 | Capreolus.Capreolus_PA1_RUN9_RF | PA1 | RUN9 | RF | ROC | 359 | 99,953 | 99,929 | 1 | 0,727 | NA 104 | Capreolus.Capreolus_PA1_RUN9_RF | PA1 | RUN9 | RF | TSS | 354 | 99,953 | 99,914 | 0,999 | 0,058 | NA 105 | Capreolus.Capreolus_PA1_RUN9_MAXENT | PA1 | RUN9 | MAXENT | ROC | 376,5 | 82,751 | 58,407 | 0,77 | 0,742 | NA 106 | Capreolus.Capreolus_PA1_RUN9_MAXENT | PA1 | RUN9 | MAXENT | TSS | 376 | 82,797 | 58,343 | 0,412 | 0,374 | NA 107 | Capreolus.Capreolus_PA1_RUN9_MAXNET | PA1 | RUN9 | MAXNET | ROC | 410,5 | 80,746 | 58,307 | 0,756 | 0,744 | NA 108 | Capreolus.Capreolus_PA1_RUN9_MAXNET | PA1 | RUN9 | MAXNET | TSS | 433 | 77,669 | 61,121 | 0,389 | 0,387 | NA 109 | Capreolus.Capreolus_PA1_RUN10_GLM | PA1 | RUN10 | GLM | ROC | 555,5 | 74,732 | 63,186 | 0,749 | 0,734 | NA 110 | Capreolus.Capreolus_PA1_RUN10_GLM | PA1 | RUN10 | GLM | TSS | 509 | 79,86 | 57,914 | 0,379 | 0,362 | NA 111 | Capreolus.Capreolus_PA1_RUN10_GBM | PA1 | RUN10 | GBM | ROC | 544,5 | 78,648 | 62,814 | 0,764 | 0,739 | NA 112 | Capreolus.Capreolus_PA1_RUN10_GBM | PA1 | RUN10 | GBM | TSS | 537 | 79,674 | 61,643 | 0,414 | 0,383 | NA 113 | Capreolus.Capreolus_PA1_RUN10_GAM | PA1 | RUN10 | GAM | ROC | 504,5 | 79,953 | 60,407 | 0,757 | 0,74 | NA 114 | Capreolus.Capreolus_PA1_RUN10_GAM | PA1 | RUN10 | GAM | TSS | 502 | 80,047 | 60,15 | 0,401 | 0,374 | NA 115 | Capreolus.Capreolus_PA1_RUN10_RF | PA1 | RUN10 | RF | ROC | 383 | 100 | 99,957 | 1 | 0,721 | NA 116 | Capreolus.Capreolus_PA1_RUN10_RF | PA1 | RUN10 | RF | TSS | 383 | 100 | 99,957 | 1 | 0,028 | NA 117 | Capreolus.Capreolus_PA1_RUN10_MAXENT | PA1 | RUN10 | MAXENT | ROC | 404,5 | 79,953 | 61,25 | 0,767 | 0,738 | NA 118 | Capreolus.Capreolus_PA1_RUN10_MAXENT | PA1 | RUN10 | MAXENT | TSS | 405 | 79,953 | 61,25 | 0,412 | 0,378 | NA 119 | Capreolus.Capreolus_PA1_RUN10_MAXNET | PA1 | RUN10 | MAXNET | ROC | 430,5 | 77,902 | 61,243 | 0,755 | 0,739 | NA 120 | Capreolus.Capreolus_PA1_RUN10_MAXNET | PA1 | RUN10 | MAXNET | TSS | 449 | 75,385 | 63,643 | 0,391 | 0,369 | NA

calib.lines <- get_calib_lines(Capreolus_single_models, PA = "allData", run = "allRun")
biomod2::plot(Capreolus_BIOMOD_data, calib.lines = calib.lines,
              plot.type = "raster", plot.eval = T, do.plot = T)

Rplot01

I really hope you can help me and give me some advice with these information, that would really help me!!! Thank you so much in advance for all your time, I wish you a great day!