search

lmfit / lmfit-py

Non-Linear Least Squares Minimization, with flexible Parameter settings, based on scipy.optimize, and with many additional classes and methods for curve fitting.
https://lmfit.github.io/lmfit-py/
Other
1.07k stars 275 forks source link

Confidence interval 'trace' in Python 3.4 fails, works v 2.7 #163

Closed kgspears closed 9 years ago

kgspears commented 9 years ago

Using Confidence Interval example (confidence2.py) in Anaconda 2.7 works, in Anaconda 3.4 fails as zip(params.keys() + ['prob'], tmp.T): in map_trace_to_names is a 2.7 style code. Python 2 to 3 covers this point.
command: ci, trace = lmfit.conf_interval(mi, sigmas=[0.68,0.95], trace=True, verbose=False)

newville commented 9 years ago

@kgspears -- Hmm, I guess we need a test for using 'trace'. If Python 2 to 3 covers this point, does that mean you have a fix for the problem?

kgspears commented 9 years ago

I am not good enough to do the fix, but they cover how the zip command can be wrapped? It would be a small project for me to learn that part of the code, but when time I would like to understand it.

Thanks, Ken

On Dec 8, 2014, at 2:23 PM, "Matt Newville" notifications@github.com<mailto:notifications@github.com> wrote:

@kgspearshttps://github.com/kgspears -- Hmm, I guess we need a test for using 'trace'. If Python 2 to 3 covers this point, does that mean you have a fix for the problem?

— Reply to this email directly or view it on GitHubhttps://github.com/lmfit/lmfit-py/issues/163#issuecomment-66170441.

newville commented 9 years ago

@kgspears -- well, I was sort of hoping you could give us a short, complete example that exposes the problem that we could run, or a traceback, or a more concrete and specific description of what you think the problem is. That is, you saw the problem that apparently none of us has seen.

What is the "this point" that "Python 2 to 3 covers"? We could guess, but you have more information than you're telling us.

kgspears commented 9 years ago

On 12/8/2014 7:48 PM, Matt Newville wrote:

@kgspears https://github.com/kgspears -- well, I was sort of hoping you could give us a short, complete example that exposes the problem that we could run, or a traceback, or a more concrete and specific description of what you think the problem is. That is, you saw the problem that apparently none of us has seen.

What is the "this point" that "Python 2 to 3 covers"? We could guess, but you have more information than you're telling us.

— Reply to this email directly or view it on GitHub https://github.com/lmfit/lmfit-py/issues/163#issuecomment-66216557.

Hi Matt, Sorry for the short reply, I was about to leave for an event and my ride just pulled up. I probably should have waited. The program example called confidence2.py with a series of plots, that you published recently, was what I used rather than my own concoction that I started with. They crashed the same under Anaconda Python 3.4. I am using lmfit 0.8.1

That section of code is: mi = lmfit.minimize(residual, p) lmfit.printfuncs.report_fit(mi.params, min_correl=0.5) ci, trace = lmfit.conf_interval(mi, sigmas=[0.68,0.95], trace=True, verbose=False) lmfit.printfuncs.report_ci(ci)

The  full traceback is:

Traceback (most recent call last):

File "", line 1, in runfile('C:/Users/ken/Documents/Python Scripts/PythonTestPrograms/Lmfit/confidence2.py', wdir='C:/Users/ken/Documents/Python Scripts/PythonTestPrograms/Lmfit')

File "G:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 580, in runfile execfile(filename, namespace)

File "G:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 48, in execfile exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)

File "C:/Users/ken/Documents/Python Scripts/PythonTestPrograms/Lmfit/confidence2.py", line 24, in ci, trace = lmfit.conf_interval(mi, sigmas=[0.68,0.95], trace=True, verbose=False)

File "G:\Anaconda\lib\site-packages\lmfit\confidence.py", line 119, in conf_interval output = ci.calc_all_ci()

File "G:\Anaconda\lib\site-packages\lmfit\confidence.py", line 203, in calc_all_ci self.minimizer.params)

File "G:\Anaconda\lib\site-packages\lmfit\confidence.py", line 131, in map_trace_to_names for para_name, values in zip(params.keys() + ['prob'], tmp.T):

TypeError: unsupported operand type(s) for +: 'KeysView' and 'list'

I looked at this function and it was using a ZIP command, which I have never used, so was not sure how it was supposed to work. The error suggested something changed in the versions so I promptly went to Python 2 to 3 web site at http://pythonhosted.org/six/ which was less helpful than http://www.diveintopython3.net/porting-code-to-python-3-with-2to3.html. This seems like an easy fix, if you know the related code that uses it.
Here is their suggestion, which they claim is fixable by the 2 to 3 utility script. Which you probably don't need.

|A.33.zip()| global function

In Python 2, the global |zip()| function took any number of sequences and returned a list of tuples. The first tuple contained the first item from each sequence; the second tuple contained the second item from each sequence; and so on. In Python 3, |zip()| returns an iterator instead of a list.

Notes Python 2 Python 3 ① |zip(a, b, c)| |list(zip(a, b, c))| ② |d.join(zip(a, b, c))| /no change/

  1. In the simplest form, you can get the old behavior of the |zip()| function by wrapping the return value in a call to |list()|, which will run through the iterator that |zip()| returns and return a real list of the results.

  2. In contexts that already iterate through all the items of a sequence (such as this call to the |join()| method), the iterator that |zip()| returns will work just fine. The |2to3| script is smart enough to detect these cases and make no change to your code.

    As you might recall, I just started programming Python in September and am not really competent to fix a code as complex as lmfit. However, the above fix might work smoothly, it really comes down to checking all the dependencies. It is amazing to see all the stuff in your code that checks for input and other consistencies.

    By the way Lmfit is a nice package and I really appreciate all that you have done to make it compatible with a number of platforms. We will use it in our group here at the U Michigan for general purpose fitting, and I was just getting started running it through paces involving confidence levels. That is a major goal for us using it in large scale Variable Projection algorithms. I have been in contact with Till Stensitzki on that problem and with his advice on an alternating least squares test model, I have a small VarPro code working that needs to be expanded and then include confidence levels via Lmfit. Of course, algorithm robustness to noise in the fitting is a major issue that he (and I )are testing.

    So, I would offer to help fix the bug if I was competent in OOP; mostly I focus on algorithms first and am adding the coding as I go along with a simple function approach.

    I might have found something else suspicious, using my own test code involving consecutive plots in the confidence interval, and I will follow it up systematically in the next week or so (it just might be my amateur programming). I thought it more important to alert you on the version inconsistency with the "trace".

    Your example, confidence_interval_2 was very, very nice (in 2.7), and I was trying to make plots about 1/10 that nice when I noticed your lovely example.

Best Regards, Ken Spears

newville commented 9 years ago

@kgspears -- thanks, just seeing the actual error message clears up that dict.keys() + list is the root of the problem -- we'll fix that.

newville commented 9 years ago

fixed with 671d0866e0693c569eb12d55fae39e80b754cbf8