ENH: add support for cosmic2 ivm

[jklenzing]

Description

Addresses #31. Downstream of #30. Will rebase into develop once that is merged.

Putting up for discussion. Known issues:

• Because all data products are stored in a single zip file, currently bypasses using an inst_id directory. Updates to pysat likely required to make this work.

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

via pytest. Requires pysat/pysat#950

Test Configuration:

• Operating system: Mac os x 10.15.7
• Version number: Python 3.8.11

Checklist:

• [x] Make sure you are merging into the develop (not main) branch
• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules
• [x] Add a note to CHANGELOG.md, summarizing the changes
• [x] Update zenodo.json file for new code contributors

[rstoneback]

You could write a download method within cosmic2_ivm that invokes the general method, and then afterwards, goes through and makes sub-directories and moves files.

[jklenzing]

You could write a download method within cosmic2_ivm that invokes the general method, and then afterwards, goes through and makes sub-directories and moves files.

I started writing this earlier, but wound up stuck on how to generate the destinations for each instrument without instantiating new instruments for each.

[rstoneback]

You could write a download method within cosmic2_ivm that invokes the general method, and then afterwards, goes through and makes sub-directories and moves files.

I started writing this earlier, but wound up stuck on how to generate the destinations for each instrument without instantiating new instruments for each.

You could search for inst_id value in the passed in data_path, swap out that area for the other instruments.... There are probably issues with the approach... I'll give it some thought on my end. 🐵

[jklenzing]

@rstoneback, I can convert from GPS time ('time') or UNIX time ('uts') for the index. Any preference?

[rstoneback]

Not on my end.

On Oct 27, 2021, at 1:46 PM, Jeff Klenzing @.***> wrote:

@rstoneback https://github.com/rstoneback, I can convert from GPS time ('time') or UNIX time ('uts') for the index. Any preference?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pysat/pysatCDAAC/pull/32#issuecomment-953212179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3N26LYRFWSWPJC4MT2WVTUJBCGRANCNFSM5GYT3FRQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[jklenzing]

I pushed the option that works (the other gives me a non-unique index). We'll need to sort out pysat/pysat#944 for the long-haul, but I suspect this branch will hang around for a while as we update standards. Gives us somethign to load and compare to in the meantime.

[jklenzing]

Noting that docs don't exist yet for this package. Adding to the issues.

[jklenzing]

Noting that this still needs appropriate acknowledgements and references

[jklenzing]

Acknowledgements based on the first results paper here: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019GL086841

For the reference, I am using the persistent DOI for the dataset which redirects here: https://www.cosmic.ucar.edu/what-we-do/cosmic-2/data The citation is based on their suggested format at the bottom of the page.

The first results paper does not include IVM results, so it's not here. We should include it when we add the GPS data in the next pull.

Thoughts? Is there anything I've missed? @aburrell, @rstoneback

[jklenzing]

TODO: add inst_id separation

[rstoneback]

I built a COSMIC-2 constellation for a seasonal analysis demo for the paper. Worked well. Data averages popped out no problem. 👍

[jklenzing]

TODO:

• [x] Fix metadata fill value
• [x] update download routine to move files by inst_id

[jklenzing]

test branch at https://github.com/pysat/pysatCDAAC/tree/nomerge/sort_files

[jklenzing]

After reviewing previous test branches and playing with syntax, here are the options (assuming we don't want more breaking changes in pysat).

Options for inst_id sorting:

• Ignore inst_id, group all files at the top level. This creates 365*6 files in each yearly directory.
• Ignore inst_id, group files by day, similar to the gps data. This creates daily subdirectories with 6 files.
• Move files after download to maintain inst_id grouping. This requires custom hard-wired code duplicating information in the instrument module, plus other instruments need to be refreshed or users will download again.
• Custom routine run elsewhere that searches all instruments in a directory and moves them accordingly. May need to be run manually (inconvenient) or at init (potentially time-consuming).

I am in favor of either of the first two, as those can be implemented in a straightforward fashion. The other two push us away from reusable code, and will make development / maintenance / usage harder. I am not convinced that maintaining an inst_id subdirectory buys us anything. The daily subdirectory creates a better consistency within the package.

[aburrell]

Move files after download to maintain inst_id grouping. This requires custom hard-wired code duplicating information in the instrument module, plus other instruments need to be refreshed or users will download again.

I am not sure I understand this. Maybe we could arrange a call at some point or put this on the agenda?

[rstoneback]

I am not convinced that maintaining an inst_id subdirectory buys us anything. The daily subdirectory creates a better consistency within the package.

If we don't maintain the directory format as specified by the user then cosmic-2 ivm will have an inconsistent structure. I see 'directory_format : {platform}/{name}/{tag}/{inst_id}' in my pysat.params.

The cosmic_gps instrument support doesn't include support for inst_ids.

Even without moving the directory structure all of cosmic2 IVMs would need a files refresh after a download.

Happy to talk more.

[jklenzing]

This branch is ready for review

[jklenzing]

Reverting back to storing files in a single directory. Maintaining an inst_id directory for this instrument involves adding multiple functions specifically for this instrument, creating maintenance problems. pysat supports alternate structures for instruments, which we are using elsewhere. I think accepting this structure here will make the code easier to maintain.