Comparisons with confidence intervals from seaborn and kmpfit (kapteyn)

[caplinje-NOAA]

Thanks so much for putting this together, this is the only simple implementation I've been able to find for confidence intervals (and way more) for generic fitting. This isn't so much an issue as it is a question. I was able to verify that the delta method agrees nearly exactly with the kmpfit confidence intervals for a few simple model/equations I was using. However, when comparing with seaborn's linear regression (which includes confidence intervals), I noticed the results for the delta method and kmpfit both did not agree with seaborns bands precisely. I'm assuming this is because the delta method (and kmpfit) are numerical in nature while seaborn uses an exact solution to confidence intervals of linear regressions. Does this make sense? Is there another reason they may disagree?

Thanks again.

[gjpelletier]

Thanks for your comments. Could you please send me a python script or notebook showing the comparisons you made between delta_method, kmpfit, and seaborn? My email is gjpelletier@gmail.com

[caplinje-NOAA]

The notebook I was working in can be found here: https://github.com/caplinje-NOAA/confidenceIntervalTesting/blob/main/TLfittingUncertainty.ipynb

Its not super clear from the plots, but the delta method and kmpfit bands overlap perfectly. Apologies for not plotting the seaborn bands directly with the others, but there are some subtle differences.

cheers

[gjpelletier]

Thanks, I started taking a look.

The seaborn.regplot function appears to use a bootstrap method according to their documentation at https://seaborn.pydata.org/generated/seaborn.regplot.html

Bootstrap is a kind of Monte Carlo method, and is a different method than the delta method.

My delta_method package has functions to use either the delta method (with the function named "delta_method"), and also for bootstrap method (with the function named "parametric_bootstrap"). These two methods usually have similar results, but they are not exactly the same. You can import both method functions as follows:

!pip install git+https://github.com/gjpelletier/delta_method.git from delta_method import delta_method, parametric_bootstrap

Examples using both the delta_method and parametric_bootstrap functions in my delta_method package are shown in the examples notebook: https://github.com/gjpelletier/delta_method/blob/main/delta_method_example.ipynb

[gjpelletier]

P.S. also note that the parametric bootstrap method will always produce different results from itself depending on how many bootstrap samples you use. For example, the seaborn.regplot default is n_boot=1000. But if you use n_boot=10000 it will give different results. You might find that as you increase n_boot that the parametric bootstrap results might get closer to the results from the delta method...

[caplinje-NOAA]

Thanks so much! All makes sense, I did not realize seaborn was using bootstrap. Ideally your methods would be added to scipy adjacent to the curve fitting methods.