search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

Fix intermediate filenames for spectral outlier detection #8734

Closed stscijgbot-jp closed 2 months ago

stscijgbot-jp commented 2 months ago

Issue JP-3719 was created on JIRA by Melanie Clarke:

Intermediate files for outlier detection on spectral data seem to have some naming problems.

Running spec3 on some NIRSpec fixed slit data (jw02361-o010_20240207t162649_spec3_00001_asn.json) and saving intermediate files from the outlier detection step, I see files saved with names like:

jw02361010001_04104_00001_nrs1_median.fits

jw02361010001_04104_00001_nrs1_outlier__outlier_s2d.fits.fits

jw02361010001_04104_00001_nrs1_outlier_i2d.fits

The "outlier_i2d" files are redundant and the "_outlier__outlier_s2d.fits.fits" are poorly named.

stscijgbot-jp commented 2 months ago

Comment by Melanie Clarke on JIRA:

Also, it looks like the outlier_i2d files are written to the working directory, not the output directory.  The _outlier__outlier_s2d.fits.fits are written to the output directory.

The i2d files appear to be intended to be temporary files, written when in_memory=False for resampling, so this issue may be related to JP-3717.

stscijgbot-jp commented 2 months ago

Comment by David Law on JIRA:

Also potentially related to https://jira.stsci.edu/browse/JP-3039 ?  That also had issues with outlier detection temporary files going to the wrong directory.

stscijgbot-jp commented 2 months ago

Comment by Melanie Clarke on JIRA:

Yes, and also JP-3311. It looks like the fixes for those issues were not complete.

stscijgbot-jp commented 2 months ago

Comment by Melanie Clarke on JIRA:

David Law - also related to these intermediate files, it looks like some recent refactoring ([https://github.com/spacetelescope/jwst/pull/8613)] removed the blot images as intermediate output for outlier_detection, for both imaging and spectral modes.

The blot images are no longer needed on disk for software reasons, but we can still save them if they are useful output for users. Does INS want them?

stscijgbot-jp commented 2 months ago

Comment by Melanie Clarke on JIRA:

Issues with intermediate files currently written are fixed by https://github.com/spacetelescope/jwst/pull/8735.

 

stscijgbot-jp commented 2 months ago

Comment by David Law on JIRA:

Melanie Clarke Thanks for flagging this to me- I'd missed that the refactoring process had disabled production of these files.  From discussion at the Sep 4 JP meeting and consultation with some key folks from the instrument teams, it sounds like we would like to retain the ability to produce such output files as optional intermediate outputs as they have appreciable scientific and testing utility.  Hopefully this will not be a difficult thing to reenable, or impact the memory savings too greatly.  I can file a dedicated ticket for this.

stscijgbot-jp commented 2 months ago

Comment by Melanie Clarke on JIRA:

Thanks for checking, David Law. Please do file a ticket for us!

stscijgbot-jp commented 2 months ago

Comment by David Law on JIRA:

Ticket https://jira.stsci.edu/browse/JP-3730 filed to reenable blot files.

stscijgbot-jp commented 2 months ago

Comment by David Law on JIRA:

Tested and working as advertised, closing.