Psf jacobian

[jeremyneveu]

The goal of this PR is to speed-up Spectractor model computation:

• [x] faster matrix multiplications
• [x] faster PSF computations
• [x] faster Jacobian computations
• [x] faster wavelength calibration

We tested JAX but model is too complex to use automatic differentiation.

[coveralls]

Pull Request Test Coverage Report for Build 6169959432

• 870 of 1026 (84.8%) changed or added relevant lines in 14 files are covered.
• 80 unchanged lines in 5 files lost coverage.
• Overall coverage increased (+5.9%) to 90.499%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
spectractor/fit/fit_spectrogram.py	66	68	97.06%
spectractor/simulation/simulator.py	65	75	86.67%
spectractor/extractor/spectrum.py	52	64	81.25%
spectractor/extractor/psf.py	121	135	89.63%
spectractor/fit/fitter.py	74	107	69.16%
spectractor/extractor/chromaticpsf.py	296	334	88.62%
spectractor/extractor/extractor.py	158	205	77.07%
<!--	Total:	870	1026	84.8%	-->

Files with Coverage Reduction	New Missed Lines	%
spectractor/extractor/extractor.py	4	83.1%
spectractor/extractor/chromaticpsf.py	6	92.03%
spectractor/simulation/image_simulation.py	7	80.29%
spectractor/fit/fitter.py	15	87.5%
spectractor/fit/fit_multispectra.py	48	79.32%
<!--	Total:	80		-->

Totals
Change from base Build 6094260689:	5.9%
Covered Lines:	6715
Relevant Lines:	7420

---

💛 - Coveralls

[jeremyneveu]

In summary, the code is faster thanks to:

• implementation of analytical derivatives
• usage of sparse matrices
• remove Minuit for calibrate_spectrum()
• computation of FWHM(lambda) on few pixels and then interpolation
• grid_search for calibrate_spectrum() only at first call If sparse_dot_mkl is used, tests on CTIO images are 10% faster.

[jeremyneveu]

Tests with AuxTel exposure dataId = {"day_obs": 20230511, "seq_num": 252, "detector": 0} at USDF in DEBUG mode (slower because of plots):	branch	total duration	1D fit iteration [s]	FFM fit iteration [s]	final cost func value
master	14m50	0.3	66	1.5263e+05
psf_jacobian	5m50	0.15	4.9	1.481e+05

[jeremyneveu]

Test with rebin=1 on dataId = {'day_obs': 20220316, 'seq_num': 330, 'detector':0}: 6GB memory usage. FFM in 22m with ~30s per FFM iterations.

[jeremyneveu]

@mfisherlevine, do you think we should move to version 2.7 or 3.0? Big parts of the core of the code has been optimized and rewritten.

[mfisherlevine]

I don't have any strong feelings as to 2.7 vs 3.0. If there are significant core algorithmic changes I can certainly see the argument for moving to 3.0, but it's totally up to you 🙂 (Also, you might want to make point-releases to iterate on this new code, in which case 3.0 also makes more sense)

[jeremyneveu]

Ok, then I switch to 3.0 :) Do you still approve the PR?