search

refunders / refund

Regression with functional data
40 stars 23 forks source link

Unreliable singular matrix errors #82

Closed kastman closed 6 years ago

kastman commented 6 years ago

Hi all,

I'm trying to troubleshoot an error in bayes_fosr that's driving me crazy, which is that during estimation some trials appear un-estimable and others are fine, but they "look" the same. The base error is this: Error in solve.default(as.numeric((A + IJ * D/2)/(b.q.sigma.me)) * sig.X + : system is computationally singular: reciprocal condition number = 7.91892e-17 which I believe is raised from this line in VB_Mult_FPCA.R L159. The collinearity of my design is quite low, and my trials all look complete (no na's, nothing strange that I can see).

Many trials can be successfully estimated, but many can't. When I look at them side by side I can't tell which is which. I've played around with the Kt and Kp parameters to try to increase resolution, but to no effect.

The odd thing is that this entire dataset estimated completely fine, with no singularity, many times repeatedly in the previous weeks, and I have absolutely no idea what changed.

Could someone (maybe @julia-wrobel or @jeff-goldsmith ) please help me understand what's causing this, and ideally how to fix it? The alternative is going through almost trial by trial to figure out which fail to estimate.

Thanks again in advance, Erik

kastman commented 6 years ago

Turns out some of my predictors were factors with empty levels - when the design matrix was being expanded those empty levels were then included and the design was singular. So the answer is simply to clean out your factors with droplevels before estimation: df$fac = droplevels(df$fac).

jeff-goldsmith commented 6 years ago

Glad this got sorted out, and thanks for following up! We'll keep this in mind as a possible data check before model fitting, although it might be something we'd want to implement across the package instead of function-by-function ...