Post Process NWP data

[peterdudfield]

Detailed Description

Currently Historic data is has some processing done in here

We could also do this processing in this repo when saving the data.

Context

• Good to have same historic and live pipelines, or shared code.

questions?

1. Is this the right place for processing NWP data?
2. Could keep this library very clean and move out the application to a different repo. This would be responsible then just of downloading the data files
3. Could move the processing into nowcasting_dataset, and process the data when loading live data - dont to this for historic
4. Could combine this repo and NWP and then we can use the same 'processing' code

[peterdudfield]

My potential solution is

• I dont like 3. to much code going into nowcasting dataset
• I dont mind expanding this a bit, maybe other people could use it
• Combine do 4. Combine repos and share processing code. There's less likely we will get things wrong further done then. To make sure we dont restrict live, we can keep the script but just load the functions in, then other processing functions can be added to historic pipeline as needed.

What do you two thing, @JackKelly and @jacobbieker

[jacobbieker]

I like 4, in terms of then it similar to Satip that has a nice wrapper for the API to just get data, and then some nice processing code as well if people want that too, and keeps it in one place.

[JackKelly]

Yeah, I agree, I like option 4 as well, for the reasons that you guys cite :slightly_smiling_face:

[peterdudfield]

I bought the code in for the NWP repo, but left NWP repo as is. I didnt want to break that repo. I think this is good enough for now.