openclimatefix / dagster-dags

Dags for running jobs on Leonardo

2 stars 0 forks source link

Add pv dag #114

Closed peterdudfield closed 5 months ago

peterdudfield commented 5 months ago

Pull Request

Description

added the code to get pv data from SS API and save to HF
tried to added assets but not jobs?
file was uploaded https://huggingface.co/datasets/openclimatefix/uk_pv/tree/main/data/2023/01/01
added monthly and yearly dags too

TODO need to deploy with SS_USER_ID= SS_API_KEY=

Q: @devsjc when its depolyed on leonardo, do cloud_archives also get loaded?

Helpes with #72

How Has This Been Tested?

tested the raw code, but the dagster bit
ran dagster locally and it works
[ ] Yes

Checklist:

[ ] My code follows OCF's coding style guidelines
[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have checked my code and corrected any misspellings

devsjc commented 5 months ago

Cloud archives do get loaded yes :)

peterdudfield commented 5 months ago

Right I think this i ready