pysat / pysatCDAAC

pysat support for CDAAC instruments
BSD 3-Clause "New" or "Revised" License
2 stars 2 forks source link

ENH: Updated cosmic_gps to use public download option #17

Closed rstoneback closed 3 years ago

rstoneback commented 3 years ago

Addresses: #13, #1 (partial)

Updates cosmic_gps to use the new public data server from CDAAC instead of the option that required authentication. Also turns on tests for cosmic_gps.

The format_str for parsing cosmic file lists was updated to parse out the ground station, version, and satellite ID. The ground station and ID parameters used 'cycle' and 'revision' keyword parameters. I also switched from the delimited filename parsing to fixed width since not all extracted parameters have the appropriate delimiter.

The check for duplicate times was updated to also performs a version check.

The time shift added to files is now constant for a given file, thus if a new version is released and downloaded, it will have the same time shift as the previous version, and will get filtered out by the version check.

rstoneback commented 3 years ago

@jklenzing - I believe there was some discussion about the sub-directory structure at one point. For this pull, I updated things to be year/doy/file. They wereyear.doy/file. It has been long enough for me that I don't remember if we talked about what to move to. What kind of improvement to directory structure are you talking about for #1?

rstoneback commented 3 years ago

Requires #14 first

jklenzing commented 3 years ago

@jklenzing - I believe there was some discussion about the sub-directory structure at one point. For this pull, I updated things to be year/doy/file. They wereyear.doy/file. It has been long enough for me that I don't remember if we talked about what to move to. What kind of improvement to directory structure are you talking about for #1?

I think this should work. As I recall, the . Was giving me trouble

rstoneback commented 3 years ago

I also tried to update the format_str to include parsing the year and day from the leading directories but that didn't work. It doesn't like leading or trailing keywords.

rstoneback commented 3 years ago

Tested new file listing function on my machine with a bunch of GPS files.

In [1]: import pysat                                                                                                                                                      
g
In [2]: gps = pysat.Instrument('cosmic', 'gps', 'ionprf', update_files=True)                                                                                              

In [3]: gps.files.files                                                                                                                                                   
Out[3]: 
2009-01-01 00:12:00.125999872    2009.001/ionPrf_C001.2009.001.00.12.G26_2013.3...
2009-01-01 00:16:00.224000000    2009.001/ionPrf_C002.2009.001.00.16.G24_2013.3...
2009-01-01 00:17:00.209000192    2009.001/ionPrf_C002.2009.001.00.17.G09_2013.3...
2009-01-01 00:27:00.109999872    2009.001/ionPrf_C001.2009.001.00.27.G10_2013.3...
2009-01-01 00:28:00.212999936    2009.001/ionPrf_C002.2009.001.00.28.G13_2013.3...
2009-01-01 00:31:00.530999808    2009.001/ionPrf_C005.2009.001.00.31.G31_2013.3...
In [4]: len(gps.files.files)                                                                                                                                              
Out[4]: 2319390
jklenzing commented 3 years ago

84.5% coverage! 👀

rstoneback commented 3 years ago

level-2 files are in. Turns out there are a whole host of level-1 files as well. For after lunch.

rstoneback commented 3 years ago

Long lunch but I've expanded support to latest level-2 and level-1 files. I left out files with unknown type or content.

rstoneback commented 3 years ago

Updates to time mangling are going to be handled in https://github.com/pysat/pysatCDAAC/pull/18. I referenced your review note about the docstring over there.