ENH: GOLD index uniqueness

pysat / pysatNASA

pysat support for NASA Instruments

BSD 3-Clause "New" or "Revised" License

21 stars 6 forks source link

ENH: GOLD index uniqueness #139

Closed jklenzing closed 1 year ago

jklenzing commented 1 year ago

Is your feature request related to a problem? Please describe. Each GOLD data file contains two instruments -- Channel A and Channel B. This breaks pysat standards since the time-series does not monotonically increase.

Describe the solution you'd like

Update the directory structure for gold to ignore 'inst_id' so we avoid the double-download issue
Use inst_id to load only part of the data

Describe alternatives you've considered Technically, someone wanting both channels would have to load the data twice. Howver, the load routine is quick.

Additional context https://github.com/pysat/pysat/issues/614

landsito commented 1 year ago

Hi, is there any plan to include tlimb and tdisk data products for GOLD data? Thanks

jklenzing commented 1 year ago

Hi @landsito, we are definitely interested in adding more GOLD data. This issue has fallen on the back burner while we sort out updates for the next version, but it's still in the queue.

jklenzing commented 1 year ago

Testing over all datasets from @landsito's branch, we have a very strong load success rate: nmax - 99.7% Success nmax - 0.3% Unique Error nmax - 0.0% Unknown Error tlimb - 100.0% Success tlimb - 0.0% Unique Error tlimb - 0.0% Unknown Error tdisk - 100.0% Success tdisk - 0.0% Unique Error tdisk - 0.0% Unknown Error o2den - 100.0% Success o2den - 0.0% Unique Error o2den - 0.0% Unknown Error

The uniqueness errors only appear in nmax data 0.3% of the time (assuming I use a sortby command to fix the non-monotonic issues). I'm going to move forward with a quick patch to fix the nmax for now. It won't necessarily replace this issue, but it will make the code much more functional.

jklenzing commented 1 year ago

NOTE: The warning message for this was dropped a few months ago when the general routine was used for init.

Alternatives to separate inst_ids:

Add nanoseconds (similar to what is used in COSMIC) to the time index for one channel. The cadence is on the order of thirty minutes, so this may be an acceptable route for fixing the last 0.3% of the data.
Add a check for non-unique data to the load routine and display a warning / error message there.

Pinging @aburrell for thoughts.

aburrell commented 1 year ago

I like the idea of nanoseconds.

jklenzing commented 1 year ago

OK. Putting this on the roadmap for 0.0.5