search

NCAR / ucomp-pipeline

Data processing pipeline for UCoMP
Other
6 stars 3 forks source link

Identify and mitigate the bad polarization images taken on 20240409 #275

Closed bberkeyU closed 6 months ago

bberkeyU commented 6 months ago

During our eclipse waves program, we got to L1 data sets that appear to have weird/bad UQV polarization images.

20240409.195647 20240409.200405

It looks like the pipeline produced these weird images from every pipeline run since April 14 (maybe 13th).

20240409 195647 ucomp 1074 l1 p3 iquv

Looking at the 195647 data, it looks like the problem occurs on the 01-average_data step, with cam0 1074.7 modulation 0 getting all of its values set to null. The raw L0 data appears to have no nulls.

Summary

We found we were processing L1/L2 files that were missing 1 or more modulation states due to removing non-redundant frames in the badFrames step. We now perform an explicit check in the averaging step; to make sure there is at least one extension with valid data for each camera-wavelength-modualtion state. If one or more is missing, the file is not prompted to the next step, and a note is made in the quality file.

bberkeyU commented 6 months ago

Both of these are wave files that are identified in the badframes file. The pipeline should set the identified bad frame to NaN's. However, with the waves file setting, a frame to nan means there isn't enough information to process the file to L1 or L2.

Over the mission, we had multiple examples of waves files that contain one or more badframes; that are appropriately not prompted to L1. It is not clear why 20240409.195647 and 20240409.200405 get promoted. We may need to add another condition to the ucomp_l1_average_data to catch all cases in which a whole flavor of data (Modulation x wavelength) isnt nan.

mgalloy commented 6 months ago

Another example of a bad frame in a waves file: 20211102.190308.

mgalloy commented 6 months ago

The pipeline found both bad files from 20240409:

...
20240520.115059 INFO: UCOMP_L1_PROCESS:  44/125 @ 1074: 20240409.195647.32.ucomp.1074.l0.fts
20240520.115101 WARN: UCOMP_L1_AVERAGE_DATA: polstate 0, camera 0, ext 4 all NaNs
20240520.115101 WARN: UCOMP_L1_STEP: ucomp_l1_average_data failed with status 1
20240520.115101 WARN: UCOMP_L1_PROCESS_FILE: skipping rest of level 1 processing for file
...
20240520.115712 INFO: UCOMP_L1_PROCESS:  57/125 @ 1074: 20240409.200405.16.ucomp.1074.l0.fts
20240520.115714 WARN: UCOMP_L1_AVERAGE_DATA: polstate 2, camera 0, ext 4 all NaNs
20240520.115714 WARN: UCOMP_L1_STEP: ucomp_l1_average_data failed with status 1
20240520.115714 WARN: UCOMP_L1_PROCESS_FILE: skipping rest of level 1 processing for file
...
mgalloy commented 6 months ago

Found the one at 20211102.190308: 

20240520.132832 INFO: UCOMP_L1_PROCESS:  19/132 @ 1074: 20211102.190308.01.ucomp.1074.l0.fts
20240520.132835 WARN: UCOMP_L1_AVERAGE_DATA: polstate 3, camera 1, ext 2 all NaNs
20240520.132835 WARN: UCOMP_L1_STEP: ucomp_l1_average_data failed with status 1
20240520.132835 WARN: UCOMP_L1_PROCESS_FILE: skipping rest of level 1 processing for file
mgalloy commented 6 months ago

I think we are finding the correct files now.