search

cars-actris / obiwan

3 stars 2 forks source link

Come together with PollyNET #1

Open HolgerPollyNet opened 1 year ago

HolgerPollyNet commented 1 year ago

The repo is here: https://github.com/NOA-ReACT/PollyXT-SCC-Pipelines

@thgeorgiou

@ihlades

@ulysses78

ihlades commented 1 year ago

I am currently restructuring obiwan to support different raw file types. Most of this is implemented already, just fixing various issues now.

Once this is done we should start on implementing PollyXT support. My idea is to use the polly_to_scc module from the PollyXT SCC Pipelines as a dependency, convert using that and then revert to the obiwan upload/download pipeline.

HolgerPollyNet commented 1 year ago

Great idea. But some issues have to be considered. See for example the discussion in the NOA repo. Also @ulysses78 can help, but he is out of office for some time now.

thgeorgiou commented 1 year ago

Hello!

In PollyXT-Pipelines we use PollyXTRepository to read Polly files, abstracting away how to join files and get data over file boundaries. This is then used to generate SCC-format files here, but this step is easy when you know the PollyXT -> SCC variable mappings. Since you already have the netCDF dependency (through scc-access), it will not be hard to implement.

Since the code above uses some project-specific concepts (e.g., we use Locations as configuration schemes for different stations), we can make changes to decouple these concepts and facilitate using the project as a library. If you have anything in mind, please let me know.