search

slaclab / lcls-tools

Python tools for LCLS: post processing data, PV handling, pulling archive settings, etc.
Apache License 2.0
2 stars 21 forks source link

Output of fit test `test_fit_gauss.py` varies with number of runs. #130

Open nneveu opened 8 months ago

nneveu commented 8 months ago

Describe the bug The tests pass and fail due to randomness in some of the function calls.

To Reproduce Steps to reproduce the behavior:

  1. Run the fit unit tests multiple times in a row: tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py

Example errors shown below.

Expected behavior The test should either fail or pass every time depending on code editions. There should be no random element or ability to pass the test if run multiple times without code edits.

Additional context Skipped these tests for now until the fitting class is updated to prevent this.

Output of fit test tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py seems to vary with # of runs. I'm looking into adjusting this test to be more consistent.

Output from running the test several times back to back:

(tools) PC101046:lcls-tools nneveu$ python -m unittest tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py ..failed returning param guesses: [4.176414857839097, 205, 100.0, 1.3921382859463656, 102.5, 50.0, 0.9064490469346526] .

Ran 3 tests in 0.417s

OK

(tools) PC101046:lcls-tools nneveu$ python -m unittest tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py ./Users/nneveu/github/lcls-tools/lcls_tools/common/data_analysis/fitting_tool.py:214: RuntimeWarning: invalid value encountered in scalar power return amp * np.exp((-abs(x - mu) * (P)) / (2 sig ** (P))) + offset ..

Ran 3 tests in 0.048s

OK

(tools) PC101046:lcls-tools nneveu$ python -m unittest tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py ...

Ran 3 tests in 0.024s

OK

(tools) PC101046:lcls-tools nneveu$ python -m unittest tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py /Users/nneveu/miniconda3/envs/tools/lib/python3.11/site-packages/scipy/optimize/_minpack_py.py:1010: OptimizeWarning: Covariance of the parameters could not be estimated warnings.warn('Covariance of the parameters could not be estimated', ...

Ran 3 tests in 0.014s

OK

(tools) PC101046:lcls-tools nneveu$ python -m unittest tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py .F.

FAIL: test_fit_tool_gaussian (tests.unit_tests.lcls_tools.common.data_analysis.test_fit_gauss.TestFitTool.test_fit_tool_gaussian)

Traceback (most recent call last): File "/Users/nneveu/github/lcls-tools/tests/unit_tests/lcls_tools/common/data_analysis/test_fit_gauss.py", line 37, in test_fit_tool_gaussian self.assertLessEqual(val["rmse"], 0.4) AssertionError: 0.8434927791146087 not less than or equal to 0.4

Ran 3 tests in 0.016s

FAILED (failures=1)

Originally posted by @nneveu in https://github.com/slaclab/lcls-tools/issues/126#issuecomment-1915818130