Error thrown in markov chain monte carlo

[ff995]

Hello again, I am getting an error message saying 'probablity function returned NaN' when running mcmc for a sample size of 135. Do you know what may cause this?

I also have some questions about how mcmc works. In your code, ln_probability is passed into EnsembleSampler in the mcmc function, is this replacing the log_prob function used in the emcee module? Also, ln_probability throws an error when I pass an array of values into it, including arrays of abscissa and ordinate values.

Desktop (please complete the following information):

• OS: Windows10
• Python Version: [e.g. 3.7]
• uravu Version: [e.g. 1.0.1]

[ff995]

Actually never mind, I have realised my mistake.

[ff995]

My apologies, but I have encountered a similar error whilst using distributions. MCMC works well with a sample of 135 normal distributions, but throws the error shown above when the sample size is 150. I am sub-sampling a much larger array so the 2 samples have few elements in common.

[arm61]

Hey @ff995 could you potential share more infomation about the problem? Or is it the code from the original issue?

[ff995]

The function I am using now is very similar to the last but I have included a screenshot of this newer one as well as the errors and the code for the conversion of ordinate values to distributions.

[arm61]

Sorry for the delay.

What are the contents of y in the lowest cell? Cause it still looks like a problem with the creation of the kernel density estimation object.

[ff995]

y is a 1D array of experimental intensities. I am first removing all values where the intensity is zero from y and then I am creating a list of Distribution objects with values in y as the mean.

[arm61]

Since you are working with normal distributions, can you try passing the y and ordinate_error=y*0.2 in to the Relationship class? This is to test if it is a problem in the creation of the Distribution-class object.

Sorry about the delay again (real life getting in the way of programming fun!)

[ff995]

Thanks for responding. Running 'mini' with an ordinate error and no distribution still works with a sample size of 135 but throws the same error with a larger sample size.

[arm61]

Is it possible that at some extrema of the parameters (guess1, etc.) there is a nonreal result to the calculate_Au3_secondary function?

[ff995]

After some probing (and asking the creator of the function), I don't think this is the case. With non-physical values the function will still return real numbers. I have also run the function with more restricted bounds but the error still persists.

[arm61]

Okay. I don't think I can debug this remotely. Would you be willing to send the relevant notebooks to me over email? (my email address should be on my profile)

[ff995]

Hi Andrew, attached is the notebook I have been using.

UravuProject (1).zip

[arm61]

Hey! There are a few functions normalise, convert_to_wavelength, etc that are called in this notebook but that aren't defined in the notebook, furthermore, the 'sec_trace.txt' is necessary to ensure everything works correctly. Would you possibly be able to send those over also? To accurately diagnose the problems, I need to be able to run everything 😄

[arm61]

These functions might be in the MMM_core.ipynb notebook?

[ff995]

Hi, I'm very sorry, here is a file containing the other notebooks. UravuProject (1).zip

[arm61]

There is still no data file ('sec_trace.txt') that I can find...

[ff995]

Very sorry (again), here is the document you're looking for. sec_trace (1).txt

[arm61]

Hey! So the good news is I know what the problem is, the bad news is there is no nice fix yet. I am gonna work a fix over the next few weeks.

In the meantime if you select less than 100 points from the trace I have been able to get something that works. This is due to the numerical instability in the kde object that is used internally within uravu.

[arm61]

Just to keep you up to date. So the problem was something deeper in the scipy library. However, I spent today working on a more robust solution and have made a pull request to the scipy package with my fix.

I am not sure how long this will take to make its way into the main scipy implementation. So I will put a workaround into uravu in the meantime and let you know when it is ready. Then once scipy have merged my change, I will silently update uravu.

[arm61]

That is the newest version of uravu up on PyPI, so if you run pip install -U uravu this should install 1.2.0 which includes the workaround until scipy merge my change.

I will close this issue for now, but if my workaround doesn't solve things please reopen.