search

mpt-network / MPTmultiverse

An R package for the comparison of MPT analysis approaches
3 stars 6 forks source link

Treatment of non-converging bootstrap models #9

Closed mariusbarth closed 6 years ago

mariusbarth commented 6 years ago

I was wondering whether it might be problematic when models that are estimated in the course of bootstrapping do not converge. Do we need to exclude these bootstrap samples from the distribution of G-squared values? Do we need to re-sample if the number of these samples exceeds some threshold?

singmann commented 6 years ago

I do not think any action in this regard is necessary. The main reason is that the non-convergence warnings usually result from non-identified parameters. What the bootstrap is supposed to help us with is exactly to identify those parameters. In case they are really non-identified, the distribution of parameter values from the bootstrap will take on essentially random values. This we can identify by looking at the distribution. Hence, excluding data sets with convergence warnings would prevent us from getting the information we are actually interested in.

singmann commented 6 years ago

Let me add one more sentence here and let me also note that I am happy to reopen this issue if anyone thinks, this is necessary. Detecting non-convergence is notoriously difficult and it is also unclear, what we should do then. We do the bootstrapping essentially to be able to ignore the warnings from the optimization algorithm. Instead, we look at an empirical assessment of identifiability based on the bootstrap distribution.