search

ricschuster / Marxan_vs_ILP_paper

https://www.biorxiv.org/content/10.1101/847632v1.abstract
0 stars 0 forks source link

Dealing with Marxan run parameters #2

Open mstrimas opened 5 years ago

mstrimas commented 5 years ago

ILP has a single parameter defining how it's run (gap to optimality), while Marxan has many: number of iterations, number of repeats, number of temperature decreases, and SPF. When exploring different scenarios (e.g. different sets or number of features, differing number of planning units), ideally we'd like to vary only one thing, but the ideal set of parameters may vary between scenario (e.g. lower number of iterations for fewer planning units) making these parameters potentially confounding. In what I've posted thus far, I've kept all parameters the same across all scenarios (SPF = 10, # iterations = 10^8, # temp decreases = 0.2 * # iterations, # repeats = 10) to avoid this confounding effect. But we may want to explore some standardized approach to calibration for each scenario.

As a side note, I think this calibration step is a huge source of confusion for new Marxan users, and avoiding that is a massive benefit of using prioritizr that should be discussed in the paper.

ricschuster commented 5 years ago

It would be great to get Matt Watts' sense of this here, because he is obviously the one, who would know the most about this. The issues with Marxan parameters was actually the reason for me to create my first Shiny app, which was a Marxan parameter calibration app (SPF, # iteration, # repeats, BLM). For the CDF case study, I would recommend we calibrate SPF and # iterations once, for the most complex problem and then keep constant.

re: side note. I completely agree and I would even argue a huge source of potential 'error', because of the subjectivity of many of these parameters, when it comes to more complex problems. I struggled with that a lot when I first started using Marxan, which is part of the reason why I'm really happy about ILP.

jeffreyhanson commented 5 years ago

Yeah, IIRC they varied the SPFs, number of iterations, and number of replicates in Hawthorne's paper. If the overall goal is compare "best possible Marxan" to ILP, then maybe we could vary the SPFs, and set the number of iterations + replicates really high for all runs (e.g. 1e+8 and 1e+3 respectively)? My impression is that most Marxan users just use the default annealing parameters.