Models can have indistinguishable criterion values, such that choosing one over the other is determined by numerical noise.
For example, in one case, during a forward search, two models with the same number of parameters were calibrated to have the same likelihood to several decimal places. Repeating the forward search 100 times resulted in 2 different trajectories through model space, occurring approximately 50/50. This is because numerical noise determined which of the two similar models were chosen at this point during the forward search.
Options to handle:
emit a warning that it is unclear which model to select, when models are very similar
create branches in the model selection, when encountering models that are within some epsilon criterion of each other or the best model so far
allow users to restart model selection at specific points, to explore trajectories that might have been chosen given different numerical noise, or were within some epsilon criterion of the best model so far
Models can have indistinguishable criterion values, such that choosing one over the other is determined by numerical noise.
For example, in one case, during a forward search, two models with the same number of parameters were calibrated to have the same likelihood to several decimal places. Repeating the forward search 100 times resulted in 2 different trajectories through model space, occurring approximately 50/50. This is because numerical noise determined which of the two similar models were chosen at this point during the forward search.
Options to handle: