DaveStrickland
commented
3 years ago Create core class ApAutoBadcols and command line script ap_auto_badcols.py. Testing shows it works well at identifying columns where all data values are elevated, even by a small amount, but it can miss bad columns where only a fraction of the column is bad.
Example of use:
ap_auto_badcol.py 20210708/cal-T20-davestrickland-CygnusLoop_x1_y1-20210708-220604-Ha-BIN1-E-300-001.fits -l DEBUG
2021-08-17 07:43:55,238 | ApAutoBadcols | INFO | Loading extension 0 of FITS file 20210708/cal-T20-davestrickland-CygnusLoop_x1_y1-20210708-220604-Ha-BIN1-E-300-001.fits
2021-08-17 07:43:55,364 | ApAutoBadcols | DEBUG | 2-D BITPIX=-32 image with 4008 columns, 2672 rows
2021-08-17 07:43:55,794 | ApAutoBadcols | DEBUG | Raw data statistics are min=-202.56, max=40335.27, median=51.70
2021-08-17 07:44:01,579 | ApAutoBadcols | INFO | Found 33 bad columns out of 4008 columns.
2021-08-17 07:44:01,581 | ApAutoBadcols | DEBUG | Diagnostics for first 40 columns and all bad column:
col, median, local_mean, local_std, nsigma, isbad?
0000, 50.62, 52.60, 1.39, 1.43, False
0001, 52.73, 52.65, 1.29, 0.06, False
0002, 51.38, 52.80, 1.27, 1.11, False
0003, 54.53, 52.93, 1.26, 1.27, False
0004, 54.11, 53.38, 1.79, 0.41, False
0005, 52.23, 53.22, 1.77, 0.56, False
0006, 52.95, 53.39, 1.60, 0.28, False
0007, 53.86, 53.35, 1.62, 0.32, False
0008, 53.97, 53.47, 1.52, 0.33, False
0009, 57.37, 53.62, 1.68, 2.24, False
0010, 51.66, 53.42, 1.73, 1.02, False
0011, 52.49, 53.53, 1.69, 0.61, False
0012, 52.26, 53.26, 1.96, 0.51, False
0013, 52.71, 53.16, 1.96, 0.23, False
0014, 56.14, 53.07, 1.94, 1.58, False
0015, 51.98, 52.87, 1.56, 0.57, False
0016, 53.39, 53.03, 1.52, 0.24, False
0017, 50.06, 53.27, 1.63, 1.98, False
0018, 52.68, 53.45, 1.61, 0.48, False
0019, 53.05, 53.55, 1.60, 0.32, False
0020, 55.13, 53.68, 1.84, 0.79, False
0021, 53.41, 54.01, 1.83, 0.33, False
0022, 55.18, 54.06, 1.82, 0.61, False
0023, 54.18, 54.32, 1.38, 0.10, False
0024, 53.86, 54.35, 1.35, 0.36, False
0025, 57.54, 54.52, 1.29, 2.34, False
0026, 55.65, 54.49, 1.28, 0.90, False
0027, 53.93, 54.66, 1.25, 0.58, False
0028, 52.95, 54.54, 1.26, 1.26, False
0029, 52.96, 54.52, 1.26, 1.24, False
0030, 54.89, 54.47, 1.30, 0.32, False
0031, 54.86, 54.16, 0.91, 0.78, False
0032, 55.22, 54.14, 0.87, 1.24, False
0033, 53.83, 54.15, 0.87, 0.36, False
0034, 54.03, 54.23, 0.79, 0.25, False
0035, 53.28, 54.24, 0.77, 1.25, False
0036, 64.56, 54.34, 0.90, 11.34, True
0037, 55.43, 54.25, 0.89, 1.33, False
0038, 54.01, 54.25, 0.90, 0.27, False
0039, 53.78, 54.48, 1.04, 0.67, False
0109, -31.62, 54.48, 1.42, 60.54, True
0282, 61.93, 53.32, 1.29, 6.65, True
0394, 56.12, 52.55, 0.54, 6.59, True
0448, 66.72, 52.63, 1.83, 7.71, True
0472, 61.38, 52.70, 1.28, 6.76, True
0486, 66.39, 51.90, 1.84, 7.87, True
0525, 61.16, 51.94, 0.98, 9.39, True
0554, 62.72, 51.68, 1.09, 10.11, True
0594, 2.37, 49.17, 2.25, 20.78, True
0751, 127.74, 51.09, 1.66, 46.23, True
0784, 59.73, 50.73, 1.23, 7.31, True
1578, 59.96, 47.97, 0.90, 13.34, True
1810, 60.00, 48.42, 1.37, 8.45, True
1989, 78.14, 47.33, 1.10, 27.95, True
1997, 56.64, 47.40, 1.06, 8.70, True
2028, 31.63, 48.48, 1.22, 13.79, True
2330, 56.69, 50.77, 0.91, 6.53, True
2484, 55.87, 51.46, 0.52, 8.51, True
2649, 62.77, 52.73, 1.14, 8.84, True
2775, 57.63, 51.68, 0.56, 10.61, True
2934, 58.53, 53.14, 0.31, 17.51, True
3051, 60.42, 51.81, 1.12, 7.72, True
3120, 96.51, 55.09, 1.75, 23.71, True
3193, 82.57, 57.09, 1.77, 14.36, True
3310, 71.91, 56.73, 1.18, 12.82, True
3441, 118.28, 54.19, 1.53, 42.00, True
3453, 108.30, 54.37, 1.12, 48.25, True
3484, 68.03, 54.54, 1.53, 8.82, True
3500, 61.92, 53.45, 1.13, 7.47, True
3514, 62.55, 53.79, 1.69, 5.17, True
3564, 62.49, 53.89, 1.43, 6.02, True
3761, 59.97, 53.87, 0.98, 6.23, True
2021-08-17 07:44:05,425 | ApAutoBadcols | INFO | Found 0 bad rows out of 2672 rows.
2021-08-17 07:44:05,427 | ApAutoBadcols | DEBUG | Diagnostics for first 40 rows and all bad row:
row, median, local_mean, local_std, nsigma, isbad?
0000, 23.52, 24.23, 0.40, 1.78, False
0001, 24.35, 24.19, 0.38, 0.44, False
0002, 24.84, 24.18, 0.36, 1.83, False
0003, 24.41, 24.19, 0.34, 0.67, False
0004, 24.17, 24.17, 0.32, 0.01, False
0005, 24.07, 24.14, 0.32, 0.24, False
0006, 23.95, 24.22, 0.25, 1.06, False
0007, 24.16, 24.21, 0.25, 0.21, False
0008, 24.19, 24.12, 0.19, 0.41, False
0009, 24.01, 24.11, 0.18, 0.53, False
0010, 23.90, 24.10, 0.18, 1.06, False
0011, 24.38, 24.12, 0.20, 1.25, False
0012, 24.25, 24.13, 0.20, 0.63, False
0013, 23.77, 24.09, 0.23, 1.39, False
0014, 24.36, 24.11, 0.25, 0.99, False
0015, 24.00, 24.16, 0.29, 0.58, False
0016, 24.39, 24.21, 0.28, 0.64, False
0017, 23.99, 24.23, 0.30, 0.81, False
0018, 23.72, 24.28, 0.33, 1.67, False
0019, 24.45, 24.36, 0.31, 0.30, False
0020, 24.59, 24.40, 0.34, 0.57, False
0021, 24.42, 24.45, 0.31, 0.11, False
0022, 24.61, 24.47, 0.31, 0.45, False
0023, 24.73, 24.59, 0.13, 1.15, False
0024, 24.69, 24.57, 0.13, 0.94, False
0025, 24.82, 24.55, 0.16, 1.69, False
0026, 24.57, 24.54, 0.16, 0.21, False
0027, 24.56, 24.55, 0.15, 0.04, False
0028, 24.42, 24.58, 0.18, 0.87, False
0029, 24.38, 24.55, 0.18, 0.89, False
0030, 24.27, 24.54, 0.18, 1.52, False
0031, 24.45, 24.49, 0.19, 0.17, False
0032, 24.55, 24.49, 0.19, 0.31, False
0033, 24.90, 24.43, 0.25, 1.88, False
0034, 24.41, 24.47, 0.27, 0.22, False
0035, 24.66, 24.48, 0.27, 0.68, False
0036, 24.17, 24.43, 0.33, 0.81, False
0037, 24.64, 24.44, 0.33, 0.59, False
0038, 23.92, 24.36, 0.40, 1.09, False
0039, 24.77, 24.28, 0.37, 1.35, False
# Auto bad columns from 20210708/cal-T20-davestrickland-CygnusLoop_x1_y1-20210708-220604-Ha-BIN1-E-300-001.fits, sigma=5.0, window_len=11
bad_columns:
- 37
- 110
- 283
- 395
- 449
- 473
- 487
- 526
- 555
- 595
- 752
- 785
- 1579
- 1811
- 1990
- 1998
- 2029
- 2331
- 2485
- 2650
- 2776
- 2935
- 3052
- 3121
- 3194
- 3311
- 3442
- 3454
- 3485
- 3501
- 3515
- 3565
- 3762
# No bad rows detected.
Generating the YaML format user-defined bad pixel file that
ApFindBadpixelsandap_find_badpix.pyuse is labor-intensive, tedious and potentially error prone.The detection of the brightest systematically bad columns and/or rows is well suited to an algorithmic solution, which should save the user's time for "by hand" input of the more subtle bad pixel pixels or rectangles.