murphyk
commented
3 years ago I'll add a citation to your paper. How do you suggest I reword sec 5.3.3 without going into all these details?
Also I noticed the following claim in the paper below: "The only admissible decision rules are Bayesian (Wald, 1992). This means that if you have a decision rule that is not Bayesian, you can improve the statistical quality of its decisions with a Bayesian alternative (Cox and Hinkley, 1979)"
I. Osband, Z. Wen, M. Asghari, M. Ibrahimi, X. Lu, and B. Van Roy, “Epistemic Neural Networks,” in NIPS, 2021 [Online]. Available: https://github.com/deepmind/enn
droy
pglpm
fortuin
Wald never proved the theorem labelled 5.3.1: "Every admissible frequent decision procedure is bayes with respect to some possibly improper prior". As stated, I'm also fairly certain that this theorem has never been proven. It certaintly wasn't shown by Wald, because I believe he may have died before improper priors were really a thing, though limits of Bayes procedures perhaps. The first reference I know for generalized Bayes is [Sacks. "Generalized Bayes solutions in estimation problems." Ann. Math. Statist. 34 751–768, 1963], although Sacks suggests the idea has been around (in the air, I suppose) for a long while. He later refers to [Samuel Karlin. "Admissibility for Estimation with Quadratic Loss." Ann. Math. Statist. 29 (2) 406 - 436, June, 1958. https://doi.org/10.1214/aoms/1177706620], who points to work in 1950/1951 by Lehmann and by Bylth (Lehmann's student) on using limits of Bayes procedures. The latter describes an approach, now known as Blyth's method, for establishing admissibility via limits of Bayes procedures. This approach is not known to be complete, even if it is sound.
Note that a theorem that "Assuming XYZ, admissible => Bayes" is not technically a complete class theorem, because, in some problems, the class of all admissible procedures in not a complete class. Admittedly, these are strange decision problems usually, though not always. Wald did prove theorems of the form "admissible => Bayes" (as well as theorems that also established that the admissible procedures were complete) but they were for problems with finite parameter spaces or under conditions like compactness of the parameter space and continuity of the risk function. Of course, these were seminal theorems.
James Berger is really the one to be credited for proving "admissible => generalized Bayes" (and some complete class problems), but these results were only for exponential family models (+ some regularity). Indeed, in general, all existing theorems of the form "admissible => [some form of Bayes]" were, until recently, laden with regularity conditions of some sort.
Haosui Duanmu and I showed recently that "a procedure is extended admissible <==> its Bayes risk is within an infinitesimal of the Bayes optimal risk, with respect to a prior that may assign infinitesimal mass". This will appear in Annals of Stats.
Our characterization, in contrast to all previous work, has no regularity conditions and makes no strong assumptions on the problem (such as boundedness of the loss). So, in conclusion, Bayes'ness is slightly weaker than admissibility, since extended admissibility says that there is no procedure that has uniformly better risk (eps better everywhere, for some eps > 0).
In summary, the result credit'd to Wald is indeed very similar to results by Wald, but it is missing very strong assumptions that were only lifted 70 years later. Berger is due some credit here, but even his result has assumptions. If you want to handle any problem, you may need a prior assigning infinitesimal mass (our work).