[Bug]: Certain spec_hw values seem to break HST S3 reduction

jbrande commented 1 year ago

FAQ check

[X] Yes, I checked the FAQ and my question has not been addressed.

Instrument

Other (any stage)

What happened?

Something goes wrong in cut_aperture when using certain values for spec_hw.

Using the first five integrations of the demo HD 209458 data, and the demo S3 WFC3 ecf, everything seems to run perfectly fine. The demo ECF comes with spec_hw=5. However, if I increase spec_hw to large enough values (apparently 31 and up), I get an error, where the data flux array being extracted from the images no longer fits in the aperture data array being returned from cut_aperture.

Error traceback output

Starting Stage 3 Reduction

Input directory: /mnt/c/Users/j670b842/Yoni/Planetary/eureka/eureka_data/hst/HD209458/test/
Output directory: /mnt/c/Users/j670b842/Yoni/Planetary/eureka/eureka_data/hst/HD209458/S3/S3_2023-08-17_wfc3_template_run1/ap31_bg5/
Using ap=31, bg=5, expand=1
Copying S3 control file

Found 6 data file(s) ending in ima.fits
# of files in scan direction 0: 3
# of files in scan direction 1: 2
Adding 0.0, 130.981 pixels to x, y centroid position for integrations related to staring-mode image #0.
Preparing reference frame 2...
Traceback (most recent call last):
  File "/mnt/c/Users/j670b842/Yoni/Planetary/eureka/eureka_data/hst/ecfs/run_eureka_wfc3.py", line 16, in <module>
    s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path)
  File "/mnt/c/Users/j670b842/Yoni/Planetary/eureka/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 237, in reduce
    meta, log = inst.preparation_step(meta, log)
  File "/mnt/c/Users/j670b842/Yoni/Planetary/eureka/Eureka/src/eureka/S3_data_reduction/wfc3.py", line 53, in preparation_step
    meta, log = get_reference_frames(meta, log)
  File "/mnt/c/Users/j670b842/Yoni/Planetary/eureka/Eureka/src/eureka/S3_data_reduction/wfc3.py", line 125, in get_reference_frames
    cut_aperture(data, meta, log)
  File "/mnt/c/Users/j670b842/Yoni/Planetary/eureka/Eureka/src/eureka/S3_data_reduction/wfc3.py", line 1004, in cut_aperture
    apdata[n] = data.flux.values[n, ap_y1:ap_y2]
ValueError: could not broadcast input array from shape (0,170) into shape (63,170)

What operating system are you using?

No response

What version of Python are you running?

No response

What Python packages do you have installed?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

kevin218 commented 1 year ago

My first suggestion is to check whether or not the guess value within cut_aperture is valid. Try printing guess, ap_y1, and ap_y2 to see if the values take you off of the edge of the detector.

jbrande commented 1 year ago

I had been doing this, actually, but somehow didn't look closely the other day. ap_y1 sometimes goes negative, if the spec_hw is larger than the current guess location. When this happens, we try to access the various data arrays as data.___.values[n, -y1:+y2], and this will always give a zero length array, which is impossible to match dimensions to any of the destination arrays.

There's got to be a smart of way of handling this properly, maybe by restricting guess to within ap_y1 of the lower image bound and ap_y2 of the upper image bound?

kevin218 commented 1 year ago

Changing guess could have other negative implications during spectral extraction. I suggest setting ap_y1 to 0 if it goes negative.

Actually, this might cause an error if the array sizes don't match.

kevin218 / Eureka