[x] Merge all Parts into single branch and care to dev/jml_v2 to ensure all original changes made it into their respective spots
Updates to database:
[x] In the event of SVO web server being down, use webbpsf bandpasses to calculate filter info for database.
[x] Added a simple create_database() function that automates db generation. This is made possible through the webbpsf_ext.imreg_tools.get_files() function.
[x] Database creation was failing for TA Confirmation images because they could not find a science observation to attach to. Instead, I think it's best to specify the coronagraphic mask (even if it's not in the header) for NIRCam TA and TA Confirmation images. This prevents association of TA files with specific sci or ref observations, which I think is preferred since they should not be associated with specific science filters, especially if TA filter was not part of the science observation (e.g., no F335M in PID 1386).
[x] Include custom NIRCam aperture name determination to fix incorrect APERNAME keyword in headers for some types of simultaneous observations.
Updates to the pipeline run_obs functions:
[x] Break out single pipeline call to a new run_single_file() function that accepts a single filename, the output directory, step dictionary, and some optional keywords for saving additional data and skipping steps.
[x] Reworked the run_obs functions to skip processing of data if it already exists, but hadn't yet been updated in database. That way running a given notebook won't unnecessarily reprocess the data. Can be overridden with overwrite keyword.
[x] Added a verbose keyword to show / hide log info during pipeline processing. If False, pipeline messages below ERROR are suppressed, but spaceKLIP-specific info messages (e.g.--> Coron1Pipeline [...] will persist.
[x] Added additional quiet keyword that removes all printed informational outputs (except for ERROR and CRITICAL) and instead only shows progress bars for Concatenation and FITS files loops.
Minor updates to stage2:
[x] remove _median.fits generated by outlier detection step if we are not explicitly saving intermediate files
Correct 1/f noise striping in stage1:
[x] Added a 1/f noise filtering algorithm that works on group level data to model the spatial 1/f striping and subtract it. Adds about 100 msec for a single 320x320 frame. Expect time to increase proportional to frame size. See issue #97 and fnoise.py for more details.
With and without 1/f noise correction for a filter in PID 1386 Obs 116:
dev/jml_v2to ensure all original changes made it into their respective spotsUpdates to database:
create_database()function that automates db generation. This is made possible through thewebbpsf_ext.imreg_tools.get_files()function.Updates to the pipeline
run_obsfunctions:run_single_file()function that accepts a single filename, the output directory, step dictionary, and some optional keywords for saving additional data and skipping steps.run_obsfunctions to skip processing of data if it already exists, but hadn't yet been updated in database. That way running a given notebook won't unnecessarily reprocess the data. Can be overridden withoverwritekeyword.verbosekeyword to show / hide log info during pipeline processing. If False, pipeline messages below ERROR are suppressed, but spaceKLIP-specific info messages (e.g.--> Coron1Pipeline [...]will persist.quietkeyword that removes all printed informational outputs (except for ERROR and CRITICAL) and instead only shows progress bars for Concatenation and FITS files loops.Minor updates to stage2:
_median.fitsgenerated by outlier detection step if we are not explicitly saving intermediate filesCorrect 1/f noise striping in stage1:
fnoise.pyfor more details.With and without 1/f noise correction for a filter in PID 1386 Obs 116: