urbach
commented
9 years ago I have to excuse, but I don't get any notification when an issue is opened here. So I simply don't realise.
We have to keep in mind that the paper by Chris is quite old by now. We have much higher statistics nowadays, and I actually seem to not need the eigenvalue replacement when I used enough bootstrap samples (~1500) to estimate the covariance matrix.
ad 1) I'm aware that I'm not exactly implementing what was proposed in the paper. I played a bit and this is what I found to be the best.
ad 2) yes, this is correct
ad 3) indeed the eigenvalue replacement is not sensible here, I'm aware of this since quite some time, but never had the time to solve it appropriately. I also wanted to pass the number of measurements separately, but this leads to major rewriting of some interfaces.
ad 4) I'm not aware of extensive studies. But I can tell that we have no problems to fit excited states coming from a GEVP with covariance matrix. Only the effective mass fit has problems sometimes, which I don't understand yet. This R implementation is also crosschecked by some members of my group using completely independent code and means to implement the covariance matrix. We so far always found almost identical results.
But I agree, we should better understand and investigate this, as it appears to become the standard now.
I think invertCovMatrix does not reproduce hep-lat/9412087 exactly. I have been looking into the issue of the undulating effective mass plateaus and the corresponding "bad" ChiSqr of the effective mass fit and hence had to look in a bit more detail at what is going on under the hood.
1) The eigenvalue replacement seems to not follow the prescription in the paper exactly. In particular (although I may be misunderstanding what happens in the construction of the inverse), it seems that the eigenvalues are not rescaled by the factor K.
2) The decision to retain an eigenvalue is not taken as in the paper but all n-sqrt(N) small eigenvalues are replaced by the average, even if some of those could be larger than the average.
3) I'm not sure it is reasonable to do blocking of the bootstrapped data for the construction of the inverse as is done for the correlated effective mass fit. Also, the number of eigenvalues which are replaced has no meaning here. Rather, in my opinion, the number of measurements should be passed as a function argument and then divided by boot.l. (currently it's the number of bootstrap samples divided by boot.l)
4) Was the inverse covariance matrix ever studied in depth for B55 where we probably have enough data to build the real thing? Is the same reduction in ChiSqr seen? (compared to using the "real" inverse covariance matrix) What about fits with explicit excited state contributions and GEVPs?
Finally: I can see that we have a bit of an issue because we are fitting exponentially decaying data which has similarly decaying variance (although of course relative variance grows with t) and thus exponentially growing inverse (co-)variance. And although multiplication with the squared distances should renormalize the resulting contribution to ChiSqr (since the squared distances are themselves exponentially decaying functions of t). It seems to me that this eigenvalue replacement results in more emphasis being put on early timeslices, which is perhaps not a bad thing but I find it a bit hard to really understand the effect this has on the resulting fit. In particular, when I have 3x170 (170 blocked measurements, ll, ls, ss) well-separated measurements and I want to fit 3x20 timeslices or so, I would expect the real inverse covariance matrix to give a good ChiSqr if the model is justified since presumably all correlations are now taken into account. However, the ChiSqr that results when I use a "real" inverse covariance matrix as the weight is generally by about a factor of 3 too large, suggesting to me that perhaps our model is wrong and we absolutely should add some excited state contribution or just do a GEVP instead... On the other hand, using the inverse covariance matrix provided by invertCovMatrix with eigenvalue replacement results in a ChiSqr which seems to justify the model.