openclimatefix / nowcasting_dataset

Prepare batches of data for training machine learning solar electricity nowcasting data
https://nowcasting-dataset.readthedocs.io/en/stable/
MIT License
24 stars 6 forks source link

Issue/option to download metadata once #710

Closed peterdudfield closed 1 year ago

peterdudfield commented 1 year ago

Pull Request

Description

Option to only download metadata once. As this is the same for eahc batch, no point downloading it again, if not needed

Bonus- some legacy data options

Fixes #

How Has This Been Tested?

normal UI, and tested with ML experiement

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

Checklist:

codecov-commenter commented 1 year ago

Codecov Report

Merging #710 (95fbf91) into main (f4f20cf) will decrease coverage by 0.25%. The diff coverage is 55.17%.

@@            Coverage Diff             @@
##             main     #710      +/-   ##
==========================================
- Coverage   93.25%   92.99%   -0.26%     
==========================================
  Files          48       48              
  Lines        3438     3454      +16     
==========================================
+ Hits         3206     3212       +6     
- Misses        232      242      +10     
Impacted Files Coverage Δ
nowcasting_dataset/dataset/batch.py 88.81% <29.41%> (-5.43%) :arrow_down:
nowcasting_dataset/filesystem/utils.py 97.59% <90.90%> (-1.16%) :arrow_down:
...ng_dataset/data_sources/metadata/metadata_model.py 93.05% <100.00%> (ø)

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more