Fisher matrix -inf loop issue

[astrogirl1]

Hi,

I am currently trying to run montepython with the bao_bossdr12 likelihood. I have added in two new parameters, which have flat logarithmic priors and I give them different starting values within the bounds before each run to trial and error and find a decent acceptance rate. I run using this line ' python montepython/MontePython.py run -p input/input_file.param -o chains/foldername -N steps --method Fisher --superupdate 20 '

Also, Reading an issue on this page(which I cannot find anymore) I made the change to thermodynamics.c line 2873 : before : class_test(dz < ppr->smallest_allowed_variation,...) *after : class_test(dz < zppr->smallest_allowed_variation,...)**

But I get the following error when trying to get Fisher matrix with/without my newly added parameters. Is this a convergence problem or something different?

Error in Class: background_init(L:745) :error in background_solve(ppr,pba);
=>background_solve(L:1727) :condition ((tau_end-tau_start)/tau_start < ppr->smallest_allowed_variation) is true; integration step: relative change in time =nan < machine precision : leads either to numerical error or infinite loop
>>>> For n_s[-1],Delta ln(L)=-1.000000e+30 using min(ln(L))=-6.433928e+00
 /!\ invalid value encountered in double_scalars
Updated stepsize. Before, after = -inf nan

Error in Class: background_init(L:745) :error in background_solve(ppr,pba);
=>background_solve(L:1727) :condition ((tau_end-tau_start)/tau_start < ppr->smallest_allowed_variation) is true; integration step: relative change in time =nan < machine precision : leads either to numerical error or infinite loop
Traceback (most recent call last):
  File "montepython/MontePython.py", line 40, in <module>
    sys.exit(run())
  File "/home/montepython_public/montepython/run.py", line 45, in run
    sampler.run(cosmo, data, command_line)
  File "/home/montepython_public/montepython/sampler.py", line 66, in run
    mcmc.chain(cosmo, data, command_line)
  File "/home/montepython_public/montepython/mcmc.py", line 325, in chain
    sampler.get_fisher_matrix(cosmo, data, command_line, C, minimum)
  File "/home/montepython_public/montepython/sampler.py", line 573, in get_fisher_matrix
    fisher_matrix, gradient = compute_fisher(data, command_line, cosmo, center, stepsize, step_matrix)
  File "/home/montepython_public/montepython/sampler.py", line 882, in compute_fisher
    (elem_k, kdiff))
  File "/home/montepython_public/montepython/sampler.py", line 1012, in compute_fisher_element
    loglike_left, diff_1, rotated_step_left = compute_fisher_step(data,command_line,cosmo,center,step_matrix,loglike_min,one,two,step_index_1,step_index_2)
  File "/home/montepython_public/montepython/sampler.py", line 1165, in compute_fisher_step
    print(">>>> For %s[%d],Delta ln(L)=%e using min(ln(L))=%e"%(name_1,int(np.sign(diff_1[step_index_1])),deltaloglike,loglike_min))
ValueError: cannot convert float NaN to integer

[dchooper]

Hi,

This might be an issue on the CLASS side. My current guess is that CLASS is failing for some reason at that point, which MP then interprets as a nan, causing the Fisher execution to fail.

Could you try running exactly the same parameters directly in CLASS?

Best, Deanna

[brinckmann]

Also, as a side note, --superupdate and -N isn't used for the Fisher method. Are you trying to use Metropolis-Hastings (for which those flags are relevant) or Fisher (I'm unsure because you talk about acceptance rate)? I would recommend the former, as the latter isn't very reliable for data analysis. If you simply want a starting covmat and that's why you're using the Fisher method you can instead let your Metropolis-Hastings run go on for a while and force the code to compute a covmat with info --want_covmat , e.g. if it doesn't do so automatically (using --update , which is on by default).

Best, Thejs

[astrogirl1]

Hi Deanna, Thanks for your reply, I tried inputting the parameters in CLASS only but somehow they are being categorised as 'unused parameters'. Even though they are iterated when I input them through MontePython. So I might have to look into it further.

Hi Thejs, Thank you for your reply. I am currently trying to get a plot like this for my MCMC data, for which I think I need a fisher matrix? I am a bit confused myself, as it says I need fisher matrix and 2d posterior distribution of Metropolis-Hasting data. Is there a way i can calculate the fisher matrix after my run instead of during it?

source : https://arxiv.org/pdf/1804.07261.pdf, page 12

[brinckmann]

The plot in question does contain both Fisher and MCMC ellipses. It was obtained by plotting both the result of a Metropolis-Hastings run and a Fisher matrix computation. The two things are separate, unrelated, processes that I chose to plot together, where I plotted them together to show that they agree fairly well. But, in general, I would say the MCMC data is sufficient on its own and does not require "validation" by a Fisher matrix computation (if anything the need for validation is the opposite, as Fisher matrix computations are notoriously unstable and can't deviate from Gaussian posteriors). If you are already doing an MCMC / Metropolis-Hastings run that should be enough for most science purposes, assuming the chains are converged.

The contours for the MCMC run (red above) are produced using the "info" mode, python montepython/MontePython.py info chains/your_chains_directory/ and don't require a Fisher matrix to be plotted. You can do that at any point while the run is on-going, at least once sufficient convergence is obtained, but note that you will usually want your minimum R-1 value of the varying cosmological and nuisance parameters to be about 0.01 or smaller for final results (this assumes at least 4 chains, if fewer you will want a smaller R-1 as it will be artificially small). You might also want to familiarize yourself with the plotting options with python montepython/MontePython.py info --help and with the option to pass a plot file with --extra plot_files/example.plot , where we give an example in plot_files/example.plot . In particular, try to play around with the smoothing options to make sure your plots are fair representations of the posterior (the red contours above were created using an older smoothing option, so your contours may look a little different for the same forecasting likelihoods and parameters. I comment a bit about the smoothing question in the example.plot file and in the analysis code montepython/analyze.py ).

Best, Thejs

[astrogirl1]

Thank you for that Thejs, that has cleared my confusion. I am now trying to run the Metropolis-Hastings algorithm to get the plots, which works. On the other hand, apparently my data has a multimodal posterior so I have now installed and setup MultiNest & PyMultiNest.

EDIT : I have now fixed this issue regarding the Multinest library. I had to re-setup and add everything to the LD_LIBRARY_PATH after I built-cmake-make in the MultiNest folder.I have to do it every time, but have made a bash script to make it easier. I am happy for this ticket to be closed.

I have followed the manuals on the internet under both the programs' Github pages and also the MontePython pages. Unfortunately, it only has worked once correctly. But now I get this error, even though I have downloaded them in the same directory where montepython is located, next to them. I have 'built' Multinest and 'python setup.py build --user' inside PyMultinest, and have also added to the LD_LIBRARY_PATH. And I run a normal command to set it off except only put -m NS after the file declaration.

Am I missing something? (My apologies if this needs to be a separate issue.)

The error :

time python montepython/MontePython.py run -m NS -p input/sdss_lrgDR7.param -o chains/bao_boss_run_test

 /!\ PyMultiNest detected but MultiNest likely not installed correctly. You can
     safely ignore this if not running with option -m NS
Running Monte Python v3.4.0

with CLASS v2.9.4

Testing likelihoods for:
 ->bao_boss_dr12, bbn

ERROR:   Could not load MultiNest library "libmultinest.so"
ERROR:   You have to build it first,
ERROR:   and point the LD_LIBRARY_PATH environment variable to it!
ERROR:   manual: http://johannesbuchner.github.com/PyMultiNest/install.html

ERROR:   Could not load MultiNest library: libmultinest.so
ERROR:   You have to build MultiNest,
ERROR:   and point the LD_LIBRARY_PATH environment variable to it!
ERROR:   manual: http://johannesbuchner.github.com/PyMultiNest/install.html

problem: libmultinest.so: cannot open shared object file: No such file or directory

real    0m0.294s
user    0m0.387s
sys     0m0.436s