Reducing stellar data

[nrsegovia]

Hi, I have recently acquired some WiFeS data and tried to process it with the pipeline. However, I get the following error:

Subtracting Overscan for OBK-622368-WiFeS-Blue-UT20240202T180035-7.fits
Traceback (most recent call last):
  File "/home/nrodriguez/wifes/workdir/reduce_data.py", line 867, in <module>
    main()
  File "/home/nrodriguez/wifes/workdir/reduce_data.py", line 849, in main
    func(obs_metadata,
  File "/home/nrodriguez/wifes/workdir/reduce_data.py", line 128, in run_overscan_sub
    pywifes.subtract_overscan(in_fn, out_fn, data_hdu=my_data_hdu)
  File "/home/nrodriguez/miniconda3/envs/pywifes/lib/python3.11/site-packages/pywifes/pywifes.py", line 727, in subtract_overscan
    subbed_data[sci[0]:sci[1],sci[2]:sci[3]] = (
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: could not broadcast input array from shape (1028,4096) into shape (2048,4096)

Looking into the code I have realized that the check for half-frames leads the script to think that my file should have a (2048,4096) shape, though it does not. This makes me wonder if the current pipeline (automation branch) is capable of reducing Stellar 1x2 images, as I might be using the wrong tool for this task. Any comments would be greatly appreciated.

As a side note, out of curiosity I tried to bypass the half-frame check by checking the y-axis range as indicated by f[data_hdu].header['DETSEC']. This allows the pipeline to proceed, but eventually hits another wall:

Traceback (most recent call last):
  File "/home/nrodriguez/wifes/workdir/reduce_data.py", line 869, in <module>
    main()
  File "/home/nrodriguez/wifes/workdir/reduce_data.py", line 851, in main
    func(obs_metadata,
  File "/home/nrodriguez/wifes/workdir/reduce_data.py", line 353, in run_wave_soln
    wifes_wsol.derive_wifes_wave_solution(wsol_in_fn, wsol_out_fn,
  File "/home/nrodriguez/miniconda3/envs/pywifes/lib/python3.11/site-packages/pywifes/wifes_wsol.py", line 1716, in derive_wifes_wave_solution
    derive_wifes_optical_wave_solution(
  File "/home/nrodriguez/miniconda3/envs/pywifes/lib/python3.11/site-packages/pywifes/wifes_wsol.py", line 1605, in derive_wifes_optical_wave_solution
    new_x, new_y, new_r = find_lines_and_guess_refs(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/nrodriguez/miniconda3/envs/pywifes/lib/python3.11/site-packages/pywifes/wifes_wsol.py", line 507, in find_lines_and_guess_refs
    mid_fit_centers = quick_arcline_fit(slitlet_data[mid_slit,:],
                                        ~~~~~~~~~~~~^^^^^^^^^^^^
IndexError: index 0 is out of bounds for axis 0 with size 0

The first few data units seem to be empty, for some reason.

[CMartinezLombilla]

Hi Nicolás, thank you for submitting your issue and for testing the pipeline. We have only tested the pipeline with classical and nod and shuffle data, so we expect to have issues with the data observed in the stellar mode. In principle, the 1x2 image size should not be a problem, but the half-frame configuration will certainly raise errors. This feedback is very valuable for us as this is something we have to test. I'll let you know how to proceed ASAP. Best, Cristina.

[nrsegovia]

Hi Cristina,

Thanks for the information 👍 I looked a bit more into this issue, and I think I found the problem. The stellar mode makes use of the "middle" of the CCD only, as pointed out by the header: DETSEC = '[1:4202,1029:3084]' / Region of detector (native pixels). This is some sort of half frame, but not the one that the pipeline expects at the top (or bottom, depending on the reference frame) of the CCD... so I would think that this needs a "third" category aside from full or half frame.

The change I did to my version of the pipeline allowed it to proceed by identifying my files as half frames, but everything breaks once the slits come into play. I believe I tracked this problem down to the half frame configuration selecting the first 6 slits from the reference list (which currently creates the empty arrays I mentioned in my previous post), while I think that in my case I would need the ones in the middle (6-12 using 0-based indexes). I tried changing this by hand, but then I had other issues with interpolating functions and array slicing, which I have yet to analyze.

[CMartinezLombilla]

Hi @nrsegovia, we've been working on this issue which turned out to be quite complex (and not previously implemented in the pipeline... just partially). We are about to merge the changes into the automation branch probably today or tomorrow, so it'll be released for everyone. In the meantime, if you have some time, it'd be good if you give it a try with your data and send us any issues. To do so, please clone/install the repo from the cml_stellar branch and run the pipeline as always.

Thank you and sorry that it took us quite a while to solve it.

[nrsegovia]

Thanks for this new update! I will try to run some tests and let you know what I get, though perhaps I'm not able to finish by tomorrow. Still, I can always re-open the issue (or create a new one) if anything does not work as intended.

[CMartinezLombilla]

Hi @nrsegovia, after quite a few tests and debugging, the stellar mode is now working in both the blue and red arms so it is available in the automation branch. I'll close this issue, but if something comes up, please let us know. Thank you for your great feedback!