Closed jklenzing closed 1 year ago
You could write a download method within cosmic2_ivm that invokes the general method, and then afterwards, goes through and makes sub-directories and moves files.
You could write a download method within cosmic2_ivm that invokes the general method, and then afterwards, goes through and makes sub-directories and moves files.
I started writing this earlier, but wound up stuck on how to generate the destinations for each instrument without instantiating new instruments for each.
You could write a download method within cosmic2_ivm that invokes the general method, and then afterwards, goes through and makes sub-directories and moves files.
I started writing this earlier, but wound up stuck on how to generate the destinations for each instrument without instantiating new instruments for each.
You could search for inst_id value in the passed in data_path, swap out that area for the other instruments.... There are probably issues with the approach... I'll give it some thought on my end. 🐵
@rstoneback, I can convert from GPS time ('time') or UNIX time ('uts') for the index. Any preference?
Not on my end.
On Oct 27, 2021, at 1:46 PM, Jeff Klenzing @.***> wrote:
@rstoneback https://github.com/rstoneback, I can convert from GPS time ('time') or UNIX time ('uts') for the index. Any preference?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pysat/pysatCDAAC/pull/32#issuecomment-953212179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3N26LYRFWSWPJC4MT2WVTUJBCGRANCNFSM5GYT3FRQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
I pushed the option that works (the other gives me a non-unique index). We'll need to sort out pysat/pysat#944 for the long-haul, but I suspect this branch will hang around for a while as we update standards. Gives us somethign to load and compare to in the meantime.
Noting that docs don't exist yet for this package. Adding to the issues.
Noting that this still needs appropriate acknowledgements and references
Acknowledgements based on the first results paper here: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2019GL086841
For the reference, I am using the persistent DOI for the dataset which redirects here: https://www.cosmic.ucar.edu/what-we-do/cosmic-2/data The citation is based on their suggested format at the bottom of the page.
The first results paper does not include IVM results, so it's not here. We should include it when we add the GPS data in the next pull.
Thoughts? Is there anything I've missed? @aburrell, @rstoneback
TODO: add inst_id separation
I built a COSMIC-2 constellation for a seasonal analysis demo for the paper. Worked well. Data averages popped out no problem. 👍
TODO:
test branch at https://github.com/pysat/pysatCDAAC/tree/nomerge/sort_files
After reviewing previous test branches and playing with syntax, here are the options (assuming we don't want more breaking changes in pysat).
Options for inst_id sorting:
I am in favor of either of the first two, as those can be implemented in a straightforward fashion. The other two push us away from reusable code, and will make development / maintenance / usage harder. I am not convinced that maintaining an inst_id subdirectory buys us anything. The daily subdirectory creates a better consistency within the package.
Move files after download to maintain inst_id grouping. This requires custom hard-wired code duplicating information in the instrument module, plus other instruments need to be refreshed or users will download again.
I am not sure I understand this. Maybe we could arrange a call at some point or put this on the agenda?
I am not convinced that maintaining an inst_id subdirectory buys us anything. The daily subdirectory creates a better consistency within the package.
If we don't maintain the directory format as specified by the user then cosmic-2 ivm will have an inconsistent structure. I see 'directory_format : {platform}/{name}/{tag}/{inst_id}' in my pysat.params.
The cosmic_gps instrument support doesn't include support for inst_ids.
Even without moving the directory structure all of cosmic2 IVMs would need a files refresh after a download.
Happy to talk more.
This branch is ready for review
Reverting back to storing files in a single directory. Maintaining an inst_id directory for this instrument involves adding multiple functions specifically for this instrument, creating maintenance problems. pysat
supports alternate structures for instruments, which we are using elsewhere. I think accepting this structure here will make the code easier to maintain.
Description
Addresses #31.
Downstream of #30. Will rebase intodevelop
once that is merged.Putting up for discussion. Known issues:
inst_id
directory. Updates to pysat likely required to make this work.Type of change
How Has This Been Tested?
via pytest. Requires pysat/pysat#950
Test Configuration:
Checklist:
develop
(notmain
) branchCHANGELOG.md
, summarizing the changes