Closed rstoneback closed 3 years ago
@jklenzing - I believe there was some discussion about the sub-directory structure at one point. For this pull, I updated things to be year/doy/file
. They wereyear.doy/file
. It has been long enough for me that I don't remember if we talked about what to move to. What kind of improvement to directory structure are you talking about for #1?
Requires #14 first
@jklenzing - I believe there was some discussion about the sub-directory structure at one point. For this pull, I updated things to be
year/doy/file
. They wereyear.doy/file
. It has been long enough for me that I don't remember if we talked about what to move to. What kind of improvement to directory structure are you talking about for #1?
I think this should work. As I recall, the . Was giving me trouble
I also tried to update the format_str to include parsing the year and day from the leading directories but that didn't work. It doesn't like leading or trailing keywords.
Tested new file listing function on my machine with a bunch of GPS files.
In [1]: import pysat
g
In [2]: gps = pysat.Instrument('cosmic', 'gps', 'ionprf', update_files=True)
In [3]: gps.files.files
Out[3]:
2009-01-01 00:12:00.125999872 2009.001/ionPrf_C001.2009.001.00.12.G26_2013.3...
2009-01-01 00:16:00.224000000 2009.001/ionPrf_C002.2009.001.00.16.G24_2013.3...
2009-01-01 00:17:00.209000192 2009.001/ionPrf_C002.2009.001.00.17.G09_2013.3...
2009-01-01 00:27:00.109999872 2009.001/ionPrf_C001.2009.001.00.27.G10_2013.3...
2009-01-01 00:28:00.212999936 2009.001/ionPrf_C002.2009.001.00.28.G13_2013.3...
2009-01-01 00:31:00.530999808 2009.001/ionPrf_C005.2009.001.00.31.G31_2013.3...
In [4]: len(gps.files.files)
Out[4]: 2319390
84.5% coverage! 👀
level-2 files are in. Turns out there are a whole host of level-1 files as well. For after lunch.
Long lunch but I've expanded support to latest level-2 and level-1 files. I left out files with unknown type or content.
Updates to time mangling are going to be handled in https://github.com/pysat/pysatCDAAC/pull/18. I referenced your review note about the docstring over there.
Addresses: #13, #1 (partial)
Updates
cosmic_gps
to use the new public data server from CDAAC instead of the option that required authentication. Also turns on tests forcosmic_gps
.The
format_str
for parsing cosmic file lists was updated to parse out the ground station, version, and satellite ID. The ground station and ID parameters used 'cycle' and 'revision' keyword parameters. I also switched from the delimited filename parsing to fixed width since not all extracted parameters have the appropriate delimiter.The check for duplicate times was updated to also performs a version check.
The time shift added to files is now constant for a given file, thus if a new version is released and downloaded, it will have the same time shift as the previous version, and will get filtered out by the version check.