emlinefit hanging for backup tile

[akremin]

For a specific tile and petal, emlinefit appears to hang during the fitting step. We should investigate whether the data is bad or whether we should improve the robustness of the code.

Night: 20240222, Tile: 41285, Petal: 5

desispec.scripts.emlinefit.main(['--coadd', '/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/41285/20240222/coadd-5-41285-thru20240222.fits', '--redrock', '/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/41285/20240222/redrock-5-41285-thru20240222.fits', '--output', '/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/41285/20240222/emline-5-41285-thru20240222.fits'])

[sbailey]

Also see ~#1984~ (CORRECTION: #1985) for another failure case with some diagnosis, but no final resolution.

Tile 40704 on 20231123 exp 206306 is another case. Mea Culpa for not reporting it at the time.

We should also systematically check for any other cases of missing emline fit files in daily (at least for the post-iron time period).

I'm associating this with Jura, since it will be a book keeping hassle if a few tiles fail to produce emline files. Heads up @araichoor .

[araichoor]

yes, I ve seen this ticket; I ll look at it this week. and thanks for tracking back the other emline-related failure! (just there s a typo, it s https://github.com/desihub/desispec/issues/1985 I believe)

[araichoor]

so I ve looked a bit at this; as for https://github.com/desihub/desispec/issues/1985, the issue is the very high ebv (reaches 4 in this tile), so the mw-transmission-corrected fluxes have extreme values like 1e10, and curve_fit() does not like that...

I m not sure what s the preferred way to handle that. I could think of:

• (1) do not fit if ebv > xxx (e.g. 2):
  • we are sure it won t change anything for >99.9% of the tiles;
  • this is "simple" code-wise (e.g. just set ivar=0 if ebv>2 in desispec.io.emline.read_emlines_inputs());
  • this "ebvmax" can be an argument of desispec.io.emline.read_emlines_inputs(), so that one can turn that off if desired;
• or (2) in the fitting function, normalize the flux (e.g. divide by np.nanmedian(fluxes[ivars != 0]), do the fitting, and rescale the fitted results by the normalization factor:
  • in theory it shouldn t change results for "regular" tiles, but I would be anxious/reluctant to change things in the fitting algorithm to solve a very marginal fraction of the cases.

i.e. I d vote for the (1), ie "do not fit if ebv > 2". @sbailey: what do you think? would you have other suggestions?

to help the discussion, few further infos.

first: the code actually runs :), but is just super-slow (takes 2h! and 1h for https://github.com/desihub/desispec/issues/1985). so we surely want to fix that.

as for the other issue, the ebv here is quite extreme. it goes up to 4, so the mw transmission-corrected flux has values as high as 1e10... (and the 1/np.sqrt(ivar) provided to curve_fit() shoots up accordingly). here s one example:

here s a plot showing how the mw transmission varies with ebv:

and here s a healpix skymap (nside=256, ie 0.05 deg2 per pixel) showing the desi footprint in yellow, and the pixels with ebv>2 in red. ignoring the ebv > 2 targets should just affect few regions of some backup tiles close to the galactic plane.

lastly, I ve parsed the iron zpix*fits catalogs: out of the 28.4M rows, I only find 81 rows with ebv > 2 (in zpix-special-backup.fits).

[sbailey]

I'm ok with masking targets with ebv>2, as long as they still produce output that is identifiable as a masked result (e.g. equivalent to a broken fiber with all wavelengths masked) and runs in a normal amount of time compared to other tiles.

[sbailey]

fixed by PR #2195