Closed rstoneback closed 3 years ago
EDIT: The error seems to be stemming from pysat.utils.time.create_datetime_index
. Stepping through the code manually, the index is corrupted after this step, with one year added to some of the data.
Following up, I've manually stepped through the code at: https://github.com/pysat/pysatCDAAC/blob/2625b77f8d5cdf5db022b746cc1d28dd3ff0cbe4/pysatCDAAC/instruments/cosmic_gps.py#L240-L263
After which all variables look as expected. Running the following line: https://github.com/pysat/pysatCDAAC/blob/2625b77f8d5cdf5db022b746cc1d28dd3ff0cbe4/pysatCDAAC/instruments/cosmic_gps.py#L265-L267
I get bad index values, with some data from 2009 interpreted as 2014, 2019 data as 2020, etc. This is running for around 36k files, including dates in 2008, 2009, 2014, 2018, 2019. Passing the last few values through as
index = pysat.utils.time.create_datetime_index(year=year[-10:],
day=day[-10:],
uts=uts[-10:])
yields correct index values. These points all error when all 36k values are passed through. I'm thinking this may be an issue with pysat if too many values are passed through this function.
Further testing report: It's not the size of the number of points, but if the files are not in sequential order. Test sequence:
year = np.array([2014, 2014, 2008])
day = np.array([1, 2, 365])
uts = np.array([0, 3600, 2700])
pysat.utils.time.create_datetime_index(year=year, day=day, uts=uts)
yields
DatetimeIndex(['2014-01-01 00:00:00', '2014-01-02 01:00:00',
'2014-12-31 00:45:00'],
dtype='datetime64[ns]', freq=None)
Since my files were downloaded out of order, this tripped up the code.
Deleting and redownloading data in sequential order (a few days from 2008, 2009, 2014), I still see this issue. I'm not certain why the files are being retrieved in a seemingly random order, but this is what is breaking everything downstream. It does not correspond to file creation date or download order.
Further testing against pysat/pysat#907 indicates that those bug fixes solve this problem here in both develop
and xarray_support
.
Closing with merge of pysat/pysat#907
@jklenzing reported that files downloaded for 2018 are reported in 2019. https://github.com/pysat/pysatCDAAC/pull/18#issuecomment-860693537
Testing by @rstoneback was not able to reproduce the issue.
Investigate what is going on with this bug.