search

pysat / pysatNASA

pysat support for NASA Instruments
BSD 3-Clause "New" or "Revised" License
21 stars 7 forks source link

BUG: bad datetimes in DE2 RPA data result in non-monotonic series #221

Open jklenzing opened 9 months ago

jklenzing commented 9 months ago

Description

Some of the data files for DE2 RPA improperly store the time epoch. This results in a non-monotonic index

To Reproduce this bug:

Steps to reproduce the behavior:

import pysat
rpa = pysat.Instrument('de2', 'rpa', strict_time_flag=False)
rpa.load(1983, 29)
rpa.data

yields

                         dataQuality      x           y            z  ionTemperature  ...       glat        glon       ilat   mlt         alt
1983-01-29 23:59:55.969            0  411.0  105.610001   310.600006          1463.0  ...  68.669998   30.740000  65.360001  3.23  350.589996
1983-01-29 23:59:59.969           60  126.0   83.610001   343.600006          1385.0  ...  68.940002   30.730000  65.610001  3.29  350.850006
1983-01-29 00:00:03.967           60  226.0   92.610001   337.600006          1357.0  ...  69.199997   30.709999  65.860001  3.32  351.109985
1983-01-29 00:00:07.967           60  279.0  103.610001   402.600006          1343.0  ...  69.459999   30.700001  66.110001  3.34  351.369995
1983-01-29 00:00:11.967            0  276.0   91.610001   440.600006          1505.0  ...  69.720001   30.680000  66.360001  3.36  351.619995
...                              ...    ...         ...          ...             ...  ...        ...         ...        ...   ...         ...
1983-01-29 19:10:17.039            0    1.0   12.610000   -95.599998          1131.0  ... -33.669998  102.120003  46.459999  1.55  252.960007
1983-01-29 19:10:20.039            0  -43.0   34.610001  -103.599998          1041.0  ... -33.470001  102.110001  46.230000  1.55  253.009995
1983-01-29 19:10:21.039            0   11.0   45.610001   -93.599998          1136.0  ... -33.400002  102.099998  46.160000  1.55  253.020004
1983-01-29 19:10:24.039            0    0.0   23.610001  4544.600098           993.0  ... -33.200001  102.089996  45.930000  1.55  253.070007
1983-01-29 19:10:25.039            0    0.0   23.610001  4587.600098          1112.0  ... -33.130001  102.089996  45.849998  1.55  253.089996

Manual inspection of the data here and on the preceeding day confirm that the first two data points should be for 1983-01-28. Loading the file with cdflib independently of pysat and plotting the raw epoch value confirms this is an issue with the file, not the datetime conversion in either package.

Test configuration

Additional context

Since this is outside of our control, an optional routine to scrub non-monotonic files. Over the 1.5 months of 1983 data inspected, this affects four data points (including the two shown here), so dropping these points is a potential fix.