[Bug]: using a single median frame for all files for curved traces results in trace not being corrected for subsequent files

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.

https://eurekadocs.readthedocs.io/

MIT License

56 stars 44 forks source link

[Bug]: using a single median frame for all files for curved traces results in trace not being corrected for subsequent files #617

Closed erinmmay closed 4 months ago

erinmmay commented 5 months ago

FAQ check

[X] Yes, I checked the FAQ and my question has not been addressed.

Instrument

NIRSpec (Stages 1-3)

What happened?

When using a single median frame for all files/batches, the trace isn't corrected after the first file/batch because it uses the median frame for this correction, which was already corrected for the first file/batch. Proposed solution: save the information on how to shift the columns so it can be applied to later files/batches just like the median frame is saved.

Error traceback output

No response

What operating system are you using?

No response

What version of Python are you running?

No response

What Python packages do you have installed?

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct

rluquer commented 4 months ago

Facing the same problem.

rluquer commented 4 months ago

A workaround (suggested by @evamariaa) is to reverse this commit: https://github.com/kevin218/Eureka/commit/6ac4a76cbd101c2cceba5cd82ce14bb26b3a77f9

erinmmay commented 4 months ago

Rather than roll this back you can either:

(1) set your nfiles in your S3 ecf to a large enough value that all segments are read in together (this requires you to have the memory/processing power to handle that) (2) set indep_batches to True (this is a new option in the S3 ecf after the update that caused this bug, it will result in S3 running like it previously ran with each file/segment getting it's own median frame. Always check the demos folder ecfs for new options when pulling down new versions of Eureka)

evamariaa commented 4 months ago

(2) set indep_batches to True (this is a new option in the S3 ecf after the update that caused this bug, it will result in S3 running like it previously ran with each file/segment getting it's own median frame. Always check the demos folder ecfs for new options when pulling down new versions of Eureka)

Thanks Erin, that's a good point, my bad! I didn't check the new ecfs, just wanted a quick workaround!