Failure in producing the covariance matrix

[facutoscano]

Hi all, I'm trying to create a new likelihood through the SPT data in Montepython (due its gaussian simplicity) changing the Cls file (see attachment). When we run the montepython code, we obtain Step XXX chain 0 : Failed to calculate covariance matrix This occurs for different number of steps (I tried 10000 and 40000) as well as changing the jumping factor to a small number (between 0 and 1). When the chain procedure finishes, it keeps only with one step and remove the others. Many thanks for your help! Cls_new.zip

[brinckmann]

Hi,

You haven't included enough information to really understand what's going on, but my best guess is that your initial guesses for the proposal distribution (sigma for each parameter) are not appropriate. The code might not manage to compute a covmat if the starting guesses aren't good enough to accumulate enough points to reasonably do so. In some cases poor start guesses will cause the chains to diverge, which makes it impossible to get something useful.

The low jumping factor indicates they're likely too large, as it's trying to shrink the proposal distribution to get more accepted points, but you should look at the acceptance rate in the .log file to confirm, which the code will try to pull towards 25%.

Did it manage to compute a covmat at any point? If so that would remove potentially a large part of the chains in a standard analysis if it happened late (since those points are not markovian). Or are they removed as burnout? You can try to analyze with --keep-non-markovianand force it to compute a new covmat with --want-covmat (warning, it will override the covmat in the directory if there is one, so make a backup if desired). You can then try to start a new run with the new covmat. Or if it managed to compute a new covmat you can try to either start over in a new directory with that covmat or you can resume with the -r flag to make the chains longer and get you more accepted points.

Best, Thejs

[facutoscano]

Hi Thejs, thank you for the quick answer! I attach the .param file, but I'm using the parameters values of Planck 2018 release and the A_PS for nuisance. No, it never compute a covmat, all the points are removed as burn-in. This is the print that appears:

_--> Scanning file chains/pr3test/2023-09-21_1000_1.txt: Removed 104 points of burn-in, and first 50 percent, keep 1 steps Step 340 chain 0 : Failed to calculate covariance matrix

new_spt_param.zip

[brinckmann]

Hi,

I suggest you look at the SPT papers and take mean values and sigmas from those instead, that should help. You might need to put a prior on tau, perhaps see what they did in their analysis. You can see an example of a tau_reio prior likelihood in montepython/likelihoods/simlow where you can change the sigma and mean value in montepython/likelihoods/simlow/simlow.data to match what they use so you can compare your results to theirs after the run.

Best, Thejs

[facutoscano]

Many thanks Thejs!