search

i4Ds / STIXCore

STIX Core functionalities
BSD 3-Clause "New" or "Revised" License
3 stars 3 forks source link

Wrong number of rows in L1 HK and L2 AUX files #357

Closed FredSchuller closed 10 months ago

FredSchuller commented 1 year ago

I found in some L2 AUX files far too many rows, because some measurements appear twice (with time delays of less than 0.01s) - see examples in attached plots. The problem is already present in the L1 hk-maxi files, see for example 2023-03-29 and 2023-03-30. This is similar to issue https://github.com/i4Ds/STIXCore/issues/237 which was closed as solved a while ago.

time_vs_row_number AUX_file_20230329 time_vs_row_number AUX_file_20230330

FredSchuller commented 1 year ago

Although the problem is present in both L1 hk-maxi and L2 ephemeris files, the number of rows is not the same: for example, for 2023-03-30, there are 3021 rows in the L1 hk-maxi file, and 2292 in the L2 file (the correct number should always be 1350).

samaloney commented 1 year ago

So I had a look at the data and in some of these file have too many data point, more than the expected 1350, anything above 1351 is suspect. Fred's example corresponds to one of the more extreme cases where the source files contain 1865, 1727 rows.

L0 data this works out to be about ~14% 😟 (170/1152) of the files. ``` # /hdd/data/stix/out/fits_v1.0.0/L0/3/25/2 import numpy as np from astropy.io import fits shapes = [] for f in files: hdul = fits.open(f) shapes.append(hdul[2].data.shape) shapes = np.array(shapes) values, counts = np.unique(shapes, return_counts=True) for v, c in zip(values, counts): print(f'Value: {v}, counts: {c}') ``` ``` Value: 143, counts: 1 Value: 202, counts: 1 Value: 277, counts: 1 Value: 283, counts: 1 Value: 327, counts: 1 Value: 329, counts: 1 Value: 331, counts: 1 Value: 353, counts: 1 Value: 369, counts: 1 Value: 386, counts: 1 Value: 390, counts: 1 Value: 459, counts: 1 Value: 528, counts: 1 Value: 578, counts: 1 Value: 583, counts: 1 Value: 589, counts: 1 Value: 612, counts: 2 Value: 622, counts: 1 Value: 627, counts: 1 Value: 634, counts: 1 Value: 638, counts: 1 Value: 652, counts: 1 Value: 654, counts: 1 Value: 669, counts: 1 Value: 670, counts: 1 Value: 696, counts: 1 Value: 697, counts: 1 Value: 709, counts: 1 Value: 724, counts: 1 Value: 766, counts: 1 Value: 783, counts: 1 Value: 810, counts: 1 Value: 969, counts: 1 Value: 999, counts: 1 Value: 1051, counts: 1 Value: 1055, counts: 1 Value: 1070, counts: 1 Value: 1078, counts: 1 Value: 1104, counts: 1 Value: 1115, counts: 1 Value: 1118, counts: 1 Value: 1122, counts: 1 Value: 1128, counts: 1 Value: 1145, counts: 1 Value: 1183, counts: 1 Value: 1205, counts: 1 Value: 1242, counts: 1 Value: 1252, counts: 1 Value: 1280, counts: 1 Value: 1295, counts: 1 Value: 1312, counts: 1 Value: 1316, counts: 2 Value: 1320, counts: 1 Value: 1324, counts: 1 Value: 1326, counts: 1 Value: 1328, counts: 1 Value: 1332, counts: 1 Value: 1334, counts: 1 Value: 1335, counts: 1 Value: 1337, counts: 1 Value: 1338, counts: 3 Value: 1341, counts: 1 Value: 1343, counts: 3 Value: 1346, counts: 2 Value: 1348, counts: 1 Value: 1349, counts: 2 Value: 1350, counts: 908 Value: 1351, counts: 17 Value: 1353, counts: 1 Value: 1355, counts: 1 Value: 1356, counts: 1 Value: 1357, counts: 1 Value: 1358, counts: 3 Value: 1359, counts: 2 Value: 1362, counts: 3 Value: 1364, counts: 2 Value: 1365, counts: 1 Value: 1368, counts: 1 Value: 1369, counts: 1 Value: 1372, counts: 1 Value: 1373, counts: 1 Value: 1375, counts: 1 Value: 1384, counts: 2 Value: 1387, counts: 1 Value: 1388, counts: 1 Value: 1389, counts: 1 Value: 1416, counts: 1 Value: 1419, counts: 1 Value: 1421, counts: 1 Value: 1424, counts: 1 Value: 1426, counts: 1 Value: 1427, counts: 1 Value: 1430, counts: 1 Value: 1434, counts: 1 Value: 1435, counts: 2 Value: 1436, counts: 1 Value: 1438, counts: 1 Value: 1443, counts: 1 Value: 1445, counts: 2 Value: 1447, counts: 1 Value: 1448, counts: 1 Value: 1449, counts: 1 Value: 1450, counts: 1 Value: 1451, counts: 2 Value: 1452, counts: 2 Value: 1456, counts: 1 Value: 1457, counts: 1 Value: 1458, counts: 1 Value: 1459, counts: 1 Value: 1469, counts: 1 Value: 1472, counts: 1 Value: 1473, counts: 1 Value: 1475, counts: 1 Value: 1478, counts: 1 Value: 1481, counts: 1 Value: 1486, counts: 1 Value: 1487, counts: 1 Value: 1494, counts: 2 Value: 1495, counts: 1 Value: 1500, counts: 1 Value: 1502, counts: 2 Value: 1506, counts: 2 Value: 1508, counts: 1 Value: 1518, counts: 2 Value: 1522, counts: 1 Value: 1523, counts: 2 Value: 1538, counts: 1 Value: 1539, counts: 1 Value: 1542, counts: 1 Value: 1546, counts: 2 Value: 1549, counts: 1 Value: 1551, counts: 1 Value: 1561, counts: 1 Value: 1563, counts: 1 Value: 1565, counts: 1 Value: 1567, counts: 2 Value: 1573, counts: 2 Value: 1588, counts: 1 Value: 1590, counts: 1 Value: 1599, counts: 1 Value: 1603, counts: 1 Value: 1605, counts: 1 Value: 1609, counts: 1 Value: 1613, counts: 1 Value: 1615, counts: 1 Value: 1626, counts: 1 Value: 1627, counts: 1 Value: 1628, counts: 1 Value: 1630, counts: 1 Value: 1633, counts: 1 Value: 1642, counts: 1 Value: 1654, counts: 1 Value: 1664, counts: 1 Value: 1680, counts: 1 Value: 1684, counts: 1 Value: 1695, counts: 1 Value: 1706, counts: 1 Value: 1709, counts: 1 Value: 1725, counts: 1 Value: 1726, counts: 2 Value: 1727, counts: 1 Value: 1729, counts: 1 Value: 1733, counts: 1 Value: 1747, counts: 1 Value: 1757, counts: 2 Value: 1758, counts: 1 Value: 1768, counts: 1 Value: 1778, counts: 1 Value: 1780, counts: 1 Value: 1789, counts: 1 Value: 1794, counts: 1 Value: 1802, counts: 1 Value: 1803, counts: 1 Value: 1814, counts: 1 Value: 1820, counts: 1 Value: 1836, counts: 1 Value: 1842, counts: 1 Value: 1843, counts: 1 Value: 1847, counts: 1 Value: 1858, counts: 1 Value: 1865, counts: 1 Value: 1881, counts: 1 Value: 1891, counts: 1 Value: 1904, counts: 1 Value: 1917, counts: 1 Value: 1919, counts: 1 Value: 1921, counts: 1 Value: 1927, counts: 1 Value: 1945, counts: 1 Value: 1949, counts: 2 Value: 1962, counts: 1 Value: 1989, counts: 1 Value: 1990, counts: 1 Value: 2001, counts: 1 Value: 2044, counts: 1 Value: 2051, counts: 1 Value: 2068, counts: 1 Value: 2217, counts: 1 ```
LB data this works out to be about ~2% (27/1152) of the files. ``` # /hdd/data/stix/out/fits_v1.0.0/LB/3/25/2 import numpy as np from astropy.io import fits shapes = [] for f in files: hdul = fits.open(f) shapes.append(hdul[2].data.shape) shapes = np.array(shapes) values, counts = np.unique(shapes, return_counts=True) for v, c in zip(values, counts): print(f'Value: {v}, counts: {c}') ``` ``` Value: 143, counts: 1 Value: 202, counts: 1 Value: 262, counts: 1 Value: 283, counts: 1 Value: 327, counts: 1 Value: 329, counts: 1 Value: 331, counts: 1 Value: 353, counts: 1 Value: 369, counts: 1 Value: 386, counts: 1 Value: 390, counts: 1 Value: 411, counts: 1 Value: 459, counts: 1 Value: 528, counts: 1 Value: 578, counts: 1 Value: 583, counts: 1 Value: 589, counts: 1 Value: 607, counts: 1 Value: 612, counts: 2 Value: 627, counts: 1 Value: 634, counts: 1 Value: 638, counts: 1 Value: 652, counts: 1 Value: 654, counts: 1 Value: 669, counts: 1 Value: 696, counts: 1 Value: 697, counts: 1 Value: 709, counts: 1 Value: 724, counts: 1 Value: 766, counts: 1 Value: 783, counts: 1 Value: 810, counts: 1 Value: 944, counts: 1 Value: 969, counts: 1 Value: 999, counts: 1 Value: 1051, counts: 1 Value: 1055, counts: 1 Value: 1070, counts: 1 Value: 1078, counts: 1 Value: 1104, counts: 1 Value: 1115, counts: 1 Value: 1118, counts: 1 Value: 1122, counts: 1 Value: 1128, counts: 1 Value: 1145, counts: 1 Value: 1183, counts: 1 Value: 1205, counts: 1 Value: 1242, counts: 1 Value: 1252, counts: 1 Value: 1280, counts: 1 Value: 1294, counts: 1 Value: 1295, counts: 1 Value: 1312, counts: 1 Value: 1316, counts: 2 Value: 1320, counts: 1 Value: 1324, counts: 1 Value: 1326, counts: 1 Value: 1328, counts: 1 Value: 1332, counts: 1 Value: 1334, counts: 1 Value: 1335, counts: 1 Value: 1338, counts: 3 Value: 1341, counts: 1 Value: 1343, counts: 3 Value: 1345, counts: 1 Value: 1346, counts: 2 Value: 1348, counts: 1 Value: 1349, counts: 2 Value: 1350, counts: 1049 Value: 1351, counts: 18 Value: 1353, counts: 1 Value: 1358, counts: 2 Value: 1359, counts: 1 Value: 1654, counts: 1 Value: 1814, counts: 1 Value: 2001, counts: 1 Value: 2044, counts: 1 Value: 2217, counts: 1 ``` LB files array(['solo_LB_stix-3-25-2_0640224000_V01.fits', 'solo_LB_stix-3-25-2_0640310400_V01.fits', 'solo_LB_stix-3-25-2_0641260800_V01.fits', 'solo_LB_stix-3-25-2_0641347200_V01.fits', 'solo_LB_stix-3-25-2_0642124800_V01.fits', 'solo_LB_stix-3-25-2_0642211200_V01.fits', 'solo_LB_stix-3-25-2_0643075200_V01.fits', 'solo_LB_stix-3-25-2_0643161600_V01.fits', 'solo_LB_stix-3-25-2_0643852800_V01.fits', 'solo_LB_stix-3-25-2_0644025600_V01.fits', 'solo_LB_stix-3-25-2_0652579200_V01.fits', 'solo_LB_stix-3-25-2_0654480000_V01.fits', 'solo_LB_stix-3-25-2_0656640000_V01.fits', 'solo_LB_stix-3-25-2_0659750400_V01.fits', 'solo_LB_stix-3-25-2_0660355200_V01.fits', 'solo_LB_stix-3-25-2_0670896000_V01.fits', 'solo_LB_stix-3-25-2_0674006400_V01.fits', 'solo_LB_stix-3-25-2_0676512000_V01.fits', 'solo_LB_stix-3-25-2_0687312000_V01.fits', 'solo_LB_stix-3-25-2_0697680000_V01.fits', 'solo_LB_stix-3-25-2_0707184000_V01.fits', 'solo_LB_stix-3-25-2_0709084800_V01.fits', 'solo_LB_stix-3-25-2_0712540800_V01.fits', 'solo_LB_stix-3-25-2_0725587200_V01.fits', 'solo_LB_stix-3-25-2_0731894400_V01.fits', 'solo_LB_stix-3-25-2_0738547200_V01.fits', 'solo_LB_stix-3-25-2_0745113600_V01.fits']

To Do

FredSchuller commented 1 year ago

And here is the list of AUX (ephemeris) files that contain too many rows (only 9 files, all in 2023 - no problem found in 2020 to 2022):

Day    Nb_rows
20230329 1612
20230330 2292
20230522 1438
20230711 1955
20230713 1576
20230714 1536
20230715 1670
20230718 1770
20230719 2947
nicHoch commented 10 months ago

addressed in #360