mdietze
commented
7 years ago Just got the following email from a list-serve that seems very relevant to this issue:
For anyone interested in expert elicitation of (computer model input) probability distributions, version 3.0 of the SHeffield ELicitation Framework (SHELF) is announced - see
This is a major upgrade to version 2.0 (released 6 years ago!), featuring: Additional extensive advice for organising and conducting SHELF elicitation workshops, in the form of several new documents. Updated SHELF 1 and SHELF 2 templates, with the six versions of SHELF 2 now consolidated into a single template. Two new SHELF 3 templates for eliciting distributions for a group of uncertain quantities. This is the first time that SHELF has included templates for multivariate elicitation. Several PowerPoint slide sets to assist with explaining the different kinds of judgements that experts are required to make, and to help them to make those judgements accurately. A new R package (https://cran.r-project.org/package=SHELF) for fitting distributions to expert judgements. Examples of an evidence dossier and completed SHELF templates.
github-actions[bot]
It would be nice to have a web-based tool in PEcAn that would help users through the process of eliciting their own priors (or the priors of other experts). This tool should follow normal social science procedures to avoid known human cognitive biases (e.g. eliciting the min and max first, and then more extreme quantiles, and then working inward to the mean/median, to avoid anchoring and adjustment). Other elicitiation software exists and should be checked out to see if it can be leveraged. Graphical feedback should be given, preferably in an interactive way. Once points on the curve have been elicited, we need to also guide users through selecting the named distribution & parameters that best matches their prior (i.e. do the fits in the background, display the options).
Once the basic tool is functional, it would be good to wrap this in code that would also set up the database record. Key items that need to be discerned is whether a prior is for a general trait or a model-specific parameter, and whether the prior is being constructed for a specific PFT or for use across PFTs. These choices would help us determine whether there is any data we can query in the database to help check the elicitation against, or whether this would be double dipping.