Methods challenge: When there is 0 events in the historical control data

[danielinteractive]

We have noticed that when there are 0 events in the historical control data the overall estimation can become unstable - in the sense that the posterior of $a$ will be very close to 0, and hence the MCMC sampler has trouble with the mixing (showing up in low ESS for $a$ but also the other parameters in the MAP model). Downstream then the result for the posterior probability of larger event rate in treatment relative to control can vary between multiple MCMC runs considerably.

To do:

• [ ] Replicate with a dummy example
• [ ] Understand better why this is a problem
• [ ] Come up with recommendations what to do in such a case
• [ ] differentiate from a situation where the historical control data is not exchangeable with current control (example: COVID) which should be handled differently
  • just put directly an informative prior on $\pi_C$ e.g. from population background rate and don't use historical control dat at all
  • The prior sample size, i.e. the addition of the two beta distribution parameters can be high (say 1000) if we believe that the trial population is consistent with the general population.
  • We might again make this more robust by mixing with a 10% uniform prior component. Here it makes sense to be much more informative with the prior compared to the usual AEs, because we can directly control the prior information in the beta prior.

[danielinteractive]

@brockk is this something you came across already too maybe?