Closed rstoneback closed 2 years ago
Somehow some commits from my other pull ended up in here. sigh
Marking this as ready for review as it is nominally complete. More accurately, the basic loading, cleaning, etc. is in there. Getting the tests to pass will be most effective after getting the previous pulls in.
Ensure time mangling, if still present, is noted in the docstring.
@jklenzing How do I add Instrument tests for a COSMIC feature keyword?
It depends on what you want to test. You can add tests under the main class. See https://github.com/pysat/pysatMissions/blob/develop/pysatMissions/tests/test_instruments.py starting at line 72. There i needed to test what happened if a custom keyword was passed through.
If you need to run the end-to-end tests, there should be a way of doing this in or after the generate_instrument_list
function. This may need upstream changes at pysat.
Thanks for the info! I think the solution you offered is enough here. I added an issue over at pysat for a potential more complete solution. https://github.com/pysat/pysat/issues/805
friendly review check @jklenzing 🐈
Sorry about that. running into local issues with this branch. Downloading 2018 files which are being labeled as 2019 in cosmic.files.files
. Not sure whether it's only here or develop as well.
Sorry about that. running into local issues with this branch. Downloading 2018 files which are being labeled as 2019 in
cosmic.files.files
. Not sure whether it's only here or develop as well.
No worries. Thanks for the bug report.
@rstoneback, looking at the other milestones for 0.1.0, we've got a way to go. Should I put together a 0.0.2 release candidate with the current changes for CEDAR? Or should we continue and release when ready?
I think a 0.0.2 release would be preferable. It'd be good to have something people can use with pysat 3.0+.
My plan is to get something out for CEDAR and I have scheduled time for pysatCDAAC next week though that does make a tight schedule. With the other pull I have up which extends altitude binning to more datasets the current package is functional across COSMIC-1 and worth putting out there.
@jklenzing Can you post an example for me? I've checked all the supported file types with downloads covering days 1-4, 2018. The file list is working out on my end.
In [39]: gps = pysat.Instrument('cosmic', 'gps', 'gfsprf', update_files=True)
In [40]: gps.download(dt.datetime(2018, 1, 1), dt.datetime(2018, 1, 4))
In [41]: gps.files.files
Out[41]:
2018-01-01 00:31:00.060300032 2018/001/gfsPrf_C006.2018.001.00.31.G03_2016.1...
2018-01-01 00:37:00.061700096 2018/001/gfsPrf_C006.2018.001.00.37.G17_2016.1...
2018-01-01 00:37:00.062200064 2018/001/gfsPrf_C006.2018.001.00.37.G22_2016.1...
2018-01-01 00:40:00.061100032 2018/001/gfsPrf_C006.2018.001.00.40.G11_2016.1...
2018-01-01 00:41:00.060100096 2018/001/gfsPrf_C006.2018.001.00.41.G01_2016.1...
The issue came up when trying to download files over a year- boundary. Something like
cosmic.download(dt.datetime(2018, 12, 30), dt.datetime(2019, 1, 4))
My files currently look like
In [13]: cosmic.files.files
Out[13]:
2019-01-01 14:20:00.060199936 2019/001/ionPrf_C006.2019.001.14.20.G02_2016.1...
2019-01-01 14:20:00.061199872 2019/001/ionPrf_C006.2019.001.14.20.G12_2016.1...
2019-01-01 14:26:00.061899776 2019/001/ionPrf_C006.2019.001.14.26.G19_2016.1...
2019-01-01 14:42:00.062299904 2019/001/ionPrf_C006.2019.001.14.42.G23_2016.1...
2019-01-01 14:48:00.060300032 2019/001/ionPrf_C006.2019.001.14.48.G03_2016.1...
...
2019-12-31 20:13:00.061500160 2018/365/ionPrf_C006.2018.365.20.13.G15_2016.1...
2019-12-31 20:24:00.062400000 2018/365/ionPrf_C006.2018.365.20.24.G24_2016.1...
2019-12-31 20:40:00.061199872 2018/365/ionPrf_C006.2018.365.20.40.G12_2016.1...
2019-12-31 20:44:00.060199936 2018/365/ionPrf_C006.2018.365.20.44.G02_2016.1...
2019-12-31 20:52:00.060600064 2018/365/ionPrf_C006.2018.365.20.52.G06_2016.1...
Thanks! That is a problem.
I'm not getting the same behavior on my system. Diagnosis time.
In [49]: gps.download(dt.datetime(2018, 12, 30), dt.datetime(2019, 1, 4))
In [55]: gps.files.files[-650:-600]
Out[55]:
2018-12-31 19:50:00.060999936 2018/365/ionPrf_C006.2018.365.19.50.G10_2016.1...
2018-12-31 20:07:00.062099968 2018/365/ionPrf_C006.2018.365.20.07.G21_2016.1...
2018-12-31 20:13:00.061500160 2018/365/ionPrf_C006.2018.365.20.13.G15_2016.1...
2018-12-31 20:24:00.062399744 2018/365/ionPrf_C006.2018.365.20.24.G24_2016.1...
2018-12-31 20:40:00.061200128 2018/365/ionPrf_C006.2018.365.20.40.G12_2016.1...
2018-12-31 20:44:00.060199936 2018/365/ionPrf_C006.2018.365.20.44.G02_2016.1...
2018-12-31 20:52:00.060600064 2018/365/ionPrf_C006.2018.365.20.52.G06_2016.1...
2019-01-01 14:20:00.060199936 2019/001/ionPrf_C006.2019.001.14.20.G02_2016.1...
2019-01-01 14:20:00.061200128 2019/001/ionPrf_C006.2019.001.14.20.G12_2016.1...
2019-01-01 14:26:00.061900032 2019/001/ionPrf_C006.2019.001.14.26.G19_2016.1...
2019-01-01 14:42:00.062299904 2019/001/ionPrf_C006.2019.001.14.42.G23_2016.1...
2019-01-01 14:48:00.060299776 2019/001/ionPrf_C006.2019.001.14.48.G03_2016.1...
2019-01-04 23:05:00.062200064 2019/004/ionPrf_C006.2019.004.23.05.G22_2016.1...
2019-01-04 23:23:00.063200000 2019/004/ionPrf_C006.2019.004.23.23.G32_2016.1...
2019-01-04 23:33:00.060999936 2019/004/ionPrf_C006.2019.004.23.33.G10_2016.1...
2019-01-04 23:35:00.063099904 2019/004/ionPrf_C006.2019.004.23.35.G31_2016.1...
2019-01-04 23:41:00.062000128 2019/004/ionPrf_C006.2019.004.23.41.G20_2016.1...
Length: 18149, dtype: object
I cleared my previous files and tried again from blank slate.
In [57]: gps.download(dt.datetime(2018, 12, 30), dt.datetime(2019, 1, 4))
In [58]: gps.files.files
Out[58]:
2018-12-30 00:04:00.062799872 2018/364/ionPrf_C006.2018.364.00.04.G28_2016.1...
2018-12-30 00:14:00.061700096 2018/364/ionPrf_C006.2018.364.00.14.G17_2016.1...
2018-12-30 00:20:00.062200064 2018/364/ionPrf_C006.2018.364.00.20.G22_2016.1...
2018-12-30 00:22:00.060100096 2018/364/ionPrf_C006.2018.364.00.22.G01_2016.1...
...
2019-01-04 23:05:00.062200064 2019/004/ionPrf_C006.2019.004.23.05.G22_2016.1...
2019-01-04 23:23:00.063200000 2019/004/ionPrf_C006.2019.004.23.23.G32_2016.1...
2019-01-04 23:33:00.061000192 2019/004/ionPrf_C006.2019.004.23.33.G10_2016.1...
2019-01-04 23:35:00.063100160 2019/004/ionPrf_C006.2019.004.23.35.G31_2016.1...
2019-01-04 23:41:00.062000128 2019/004/ionPrf_C006.2019.004.23.41.G20_2016.1...
Length: 891, dtype: object
Just tried a few tests where I delete all files and start from scratch. Happened in both cases:
pysat -- develop branch pysatCDAAC -- this branch and develop branch pandas -- 1.2.3 xarray -- 0.18.0
I was behind on my packages in comparison. I cloned my environment, upgraded xarray and pandas to the same version you have, and in another round upgraded to the absolute latest. I still get the correct file dates and times in either case....
What about numpy version?
numpy 1.19.0
Installed a close numpy, as well as the latest, no dice. Files keep working for me.
In [1]: import pysat
import da^[[A
In [2]: import datetime as dt
In [3]: gps = pysat.Instrument('cosmic', 'gps', 'ionprf', update_files=True)
In [4]: gps.files.files
Out[4]: Series([], dtype: object)
In [5]: gps.download(dt.datetime(2018, 12, 30), dt.datetime(2019, 1, 4))
In [6]: gps.files.files
Out[6]:
2018-12-30 00:04:00.062799872 2018/364/ionPrf_C006.2018.364.00.04.G28_2016.1...
2018-12-30 00:14:00.061700096 2018/364/ionPrf_C006.2018.364.00.14.G17_2016.1...
2018-12-30 00:20:00.062200064 2018/364/ionPrf_C006.2018.364.00.20.G22_2016.1...
2018-12-30 00:22:00.060100096 2018/364/ionPrf_C006.2018.364.00.22.G01_2016.1...
2018-12-30 00:33:00.061400064 2018/364/ionPrf_C006.2018.364.00.33.G14_2016.1...
...
2019-01-04 23:05:00.062200064 2019/004/ionPrf_C006.2019.004.23.05.G22_2016.1...
2019-01-04 23:23:00.063200000 2019/004/ionPrf_C006.2019.004.23.23.G32_2016.1...
2019-01-04 23:33:00.061000192 2019/004/ionPrf_C006.2019.004.23.33.G10_2016.1...
2019-01-04 23:35:00.063100160 2019/004/ionPrf_C006.2019.004.23.35.G31_2016.1...
2019-01-04 23:41:00.062000128 2019/004/ionPrf_C006.2019.004.23.41.G20_2016.1...
Length: 891, dtype: object
In [7]: import numpy as np
In [8]: np.__version__
Out[8]: '1.19.2'
I did try to specifically get version 1.19 for numpy but specifying numpy=1.19.0 doesn't work and numpy=1.19 gets me numpy v 1.19.2. This is via conda.
I'm on python 3.7.6. What about netCDF4 version?
python 3.8.2, netCDF4 1.5.6
OK! I got through a fresh install and files are still looking good on my end. I don't have any other immediate ideas for things to try on my end. Open to suggestions of course and I'll give it some thought while poking at other issues.
Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:56:27)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.24.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import pysat
In [2]: gps = pysat.Instrument('cosmic', 'gps', 'ionprf', update_files=True)
In [3]: gps.files.files
Out[3]: Series([], dtype: object)
In [4]: import datetime as dt
In [5]: gps.download(dt.datetime(2018, 12, 30), dt.datetime(2019, 1, 4))
In [6]: gps.files.files
Out[6]:
2018-12-30 00:04:00.062799872 2018/364/ionPrf_C006.2018.364.00.04.G28_2016.1...
2018-12-30 00:14:00.061700096 2018/364/ionPrf_C006.2018.364.00.14.G17_2016.1...
2018-12-30 00:20:00.062200064 2018/364/ionPrf_C006.2018.364.00.20.G22_2016.1...
2018-12-30 00:22:00.060100096 2018/364/ionPrf_C006.2018.364.00.22.G01_2016.1...
2018-12-30 00:33:00.061400064 2018/364/ionPrf_C006.2018.364.00.33.G14_2016.1...
...
2019-01-04 23:05:00.062200064 2019/004/ionPrf_C006.2019.004.23.05.G22_2016.1...
2019-01-04 23:23:00.063200000 2019/004/ionPrf_C006.2019.004.23.23.G32_2016.1...
2019-01-04 23:33:00.061000192 2019/004/ionPrf_C006.2019.004.23.33.G10_2016.1...
2019-01-04 23:35:00.063100160 2019/004/ionPrf_C006.2019.004.23.35.G31_2016.1...
2019-01-04 23:41:00.062000128 2019/004/ionPrf_C006.2019.004.23.41.G20_2016.1...
Length: 891, dtype: object
I created issue #24 to track the download and file issue. That bug is out of scope for this particular pull since no changes have been made to those lines here.
OK, the date corruption is still messing with this, but I've tried to fix it as best as I can. Deleted all data, downloaded only Jan 1 2014 data for ionprf.
Here's the data loaded:
<xarray.Dataset>
Dimensions: (time: 1041, RO: 672)
Coordinates:
* time (time) datetime64[ns] 2014-01-01T00:07:28.000324096 ......
MSL_alt (time, RO) float64 76.56 78.86 81.15 83.45 ... nan nan nan
GEO_lat (time, RO) float64 23.49 23.52 23.54 23.56 ... nan nan nan
GEO_lon (time, RO) float64 136.9 136.8 136.8 136.8 ... nan nan nan
OCC_azi (time, RO) float64 -112.0 -112.1 -112.2 ... nan nan nan
Dimensions without coordinates: RO
Data variables: (12/53)
occ_id (time) float64 1.0 nan 1.0 nan nan ... 1.0 nan 1.0 nan 1.0
fiducial_id (time) object '' nan '' nan nan '' ... '' '' nan '' nan ''
reference_sat_id (time) float64 0.0 nan 0.0 nan nan ... 0.0 nan 0.0 nan 0.0
occulting_sat_id (time) float64 32.0 nan 26.0 nan nan ... nan 25.0 nan 17.0
year (time) float64 2.014e+03 nan 2.014e+03 ... nan 2.014e+03
month (time) float64 1.0 nan 1.0 nan nan ... 1.0 nan 1.0 nan 1.0
... ...
parmsfile (time) object 'parms8' nan 'parms8' ... nan 'parms8'
center (time) object 'UCAR/CDAAC' nan ... nan 'UCAR/CDAAC'
mission (time) object 'COSMIC' nan 'COSMIC' ... nan 'COSMIC'
creation_time (time) object '13-SEP-14 04:01' nan ... '13-SEP-14 04:02'
TEC_cal (time, RO) float64 200.7 202.5 204.4 206.4 ... nan nan nan
ELEC_dens (time, RO) float64 nan nan nan nan nan ... nan nan nan nan
I'm also getting a user warning from pysat about metadata, which is presumably because we are converting attributes to variables when we sum across files. Not sure why so much data is being dropped.
In [9]: gps.load(2018, 365)
<ipython-input-9-17f90e0af544>:1: UserWarning: Metadata set to defaults, as they were missing in the Instrument
gps.load(2018, 365)
In [10]: gps.data
Out[10]:
<xarray.Dataset>
Dimensions: (RO: 490, time: 95)
Coordinates:
* time (time) datetime64[ns] 2018-12-31T00:04:25.000145920 ......
MSL_alt (time, RO) float64 52.67 55.37 58.08 60.77 ... nan nan nan
GEO_lat (time, RO) float64 -36.9 -36.92 -36.93 ... nan nan nan
GEO_lon (time, RO) float64 -137.6 -137.7 -137.7 ... nan nan nan
OCC_azi (time, RO) float64 141.1 141.1 141.1 141.1 ... nan nan nan
Dimensions without coordinates: RO
Data variables: (12/53)
occ_id (time) float64 0.0 0.0 nan nan nan ... 0.0 0.0 0.0 nan 0.0
fiducial_id (time) object ' ' ' ' nan nan ... ' ' nan ' '
reference_sat_id (time) float64 -999.0 -999.0 nan nan ... -999.0 nan -999.0
occulting_sat_id (time) float64 14.0 31.0 nan nan nan ... 24.0 12.0 nan 6.0
year (time) float64 2.018e+03 2.018e+03 nan ... nan 2.018e+03
month (time) float64 12.0 12.0 nan nan ... 12.0 12.0 nan 12.0
... ...
parmsfile (time) object 'parms8' 'parms8' nan ... nan 'parms8'
center (time) object 'UCAR/CDAAC' 'UCAR/CDAAC' ... 'UCAR/CDAAC'
mission (time) object 'COSMIC' 'COSMIC' nan ... nan 'COSMIC'
creation_time (time) object '06-APR-19 01:46' ... '06-APR-19 01:46'
TEC_cal (time, RO) float64 123.9 124.2 124.3 124.6 ... nan nan nan
ELEC_dens (time, RO) float64 1.687e+05 1.783e+05 ... nan nan
I'm also getting nans in occ_id and some others. When I check MSL_alt
though all the starting altitudes are valid.
In [15]: gps[:, 0, 'MSL_alt']
Out[15]:
<xarray.DataArray 'MSL_alt' (time: 95)>
array([52.66825104, 48.42986679, 58.19499207, 58.49398804, 58.96940231,
46.29630661, 52.10069656, 63.45686722, 43.24747467, 56.62965012,
44.27114487, 56.93402863, 54.12645721, 63.38557816, 55.27773666,
50.20518494, 55.63033676, 37.34650421, 64.33950806, 55.44276428,
53.65758514, 41.82579422, 51.86764145, 36.30073547, 39.68922424,
55.43086624, 56.45871735, 43.15644836, 45.46442413, 50.55051804,
64.1120224 , 48.71077347, 54.18301392, 57.50118256, 57.89498138,
53.99144363, 47.18326569, 71.00965881, 54.10379791, 37.93962479,
48.18090439, 61.35616684, 60.07112885, 57.65946579, 67.86323547,
53.03757477, 42.25401306, 55.38709641, 47.97056198, 43.74263382,
58.46900558, 43.55466843, 35.29411697, 58.04468536, 38.9177475 ,
58.56344223, 46.8060379 , 52.69259644, 62.12597656, 53.95124054,
59.46157455, 46.85376358, 45.90418625, 66.72284698, 39.32769394,
54.50231552, 44.27587128, 49.54469299, 64.52571106, 72.15966034,
41.98462677, 40.15797806, 51.29039764, 56.70417023, 62.16763306,
40.00085068, 58.69358063, 47.50477219, 52.33638 , 56.17718506,
44.95435715, 56.55222321, 43.51290131, 56.54217911, 43.90591049,
59.87452698, 47.56930161, 57.19068146, 39.97000504, 38.92786026,
41.8406868 , 53.08489227, 66.14324188, 54.96273422, 48.13269424])
Coordinates:
* time (time) datetime64[ns] 2018-12-31T00:04:25.000145920 ... 2018-12-...
MSL_alt (time) float64 52.67 48.43 58.19 58.49 ... 53.08 66.14 54.96 48.13
GEO_lat (time) float64 -36.9 -57.23 -61.48 -39.93 ... -36.68 -26.27 -2.639
GEO_lon (time) float64 -137.6 -95.67 -6.29 -15.29 ... 44.23 54.11 72.18
OCC_azi (time) float64 141.1 86.54 39.58 13.32 ... 86.59 13.06 14.3 31.85
I'm going to have to come back to this. I've got some checkout work to do before 8/31. I'd expect some nans in the coordinate variables near the end. I had to extend all profiles to the longest profile length for the day.
Initial checks using the pysat branch at pysat/pysat#913:
File issues are solved. Loaded 'ionprf'
for 2008 (Jan 1, Jan 31, Dec 30, Dec 31) and inspected total points versus files. Dates chosen to span leap year issues. When loaded with clean_level='none'
, points match in all cases.
def num_files(inst, year, month, day):
ind1 = inst.files.files.index.year == year
ind2 = inst.files.files.index.month == month
ind3 = inst.files.files.index.day == day
print(sum(ind1 & ind2 & ind3))
Verified this loads as expected for 'scnlvl1'
for the same dates.
Going to start working on a branch downstream of this. Recommend we merge this is in to develop and treat other major changes as new branches.
Thanks for the reminder that this was already ready already 🐒
Addresses #1 by including support for loading COSMIC GPS data into xarray. Full testing of this pull requires incorporating existing pulls, namely #14 and #17, as well as updates to GitHub actions and such :)