search

NCAR / CUPiD

CUPiD is a “one stop shop” that enables and integrates timeseries file generation, data standardization, diagnostics, and metrics from all CESM components.
https://ncar.github.io/CUPiD/
Apache License 2.0
25 stars 24 forks source link

Simultaneous timeseries generation for multiple history files types within a model component #148

Open jshaw35 opened 1 month ago

jshaw35 commented 1 month ago

Is your feature request related to a problem? Please describe. I am currently using CUPiD for timeseries generation. The flexibility and ease-of-use is awesome, but I can only process a single history file type from a model component at a time (e.g. not both monthly and daily CAM outputs).

Describe the solution you'd like I have made some minor changes to cupid/run.py that allow users to specify multiple history file strings and data variable lists for each model component (in a config.yml file). I wasn't sure if these changes were compatible with other CUPiD functionality that I haven't used and wanted to check.

Describe alternatives you've considered It is less convenient but still totally functional to create a config.yml file for each history file type to post-process (e.g. h0, h1, etc).

Additional context N/A

AgentOxygen commented 1 month ago

The GenTS implementation I developed for #143 processes all of the history file types by default, ignoring the history file string parameter. I was looking into adding a parameter to filter out undesired history files, but haven't gotten that far yet. The current version might be practical enough for your use case though

jshaw35 commented 1 month ago

Hi Cameron, thanks for the heads up on the GenTS implementation! Being able to specify history files and variables is definitely helpful for time and storage use in my applications. I will check out your implementation. I am also interested in what CUPiD users/developers think about the proposed functionality and if it would be relevant to the broader user base.

mnlevy1981 commented 1 month ago

@jshaw35 The API for generating time series is a work in progress, but @TeaganKing and I are hoping to come up with a redesign that would provide more control over the time period covered by each file as well as support for variables in multiple streams. We'll start that work next week, but I don't have a good estimate for when it will be ready.

jshaw35 commented 1 month ago

@mnlevy1981 thanks for the update! I have added handling of multiple streams and some simplistic output chunking for my own use. My changes are on this fork: https://github.com/jshaw35/CUPiD feel free to use them if it is helpful!