mintproject / MINT-Transformation

MIT License
3 stars 1 forks source link

Implement streaming and gldas pipelines #74

Closed raunaqtri1 closed 4 years ago

raunaqtri1 commented 4 years ago

Adapters can now be of 3 types:

Pipeline implements streaming using asynchronous programming and considers each adapter to be Async Generator. To support Regular and Generator adapters, it uses a DefaultWrapper Async Generator Adapter.

Modified config files schema to allow root-level pipeline inputs dictionary, which can be accessed anywhere in the config file using $$.<inputs dict key> Config files are now also allowed to have "null" values for adapter inputs which would result in skipping that particular input.

Added 2 example config files average_daily_all_variables.yml and average_monthly_precipitation_rate.yml to demonstrate all the new features. Also introduced DcatRangeStream, DcatVariableStream and CroppingTransWrapper for GLDAS pipelines.

minhptx commented 4 years ago

@raunaqtri1 After this being merged, please update the Wiki accordingly. @binh-vu Please update the MINT release page regarding Raunaq's PR