search

openclimatefix / nwp-consumer

Microservice for consuming NWP data.
9 stars 3 forks source link

new file to s3 even no new data #150

Open peterdudfield opened 4 months ago

peterdudfield commented 4 months ago

Describe the bug

There seems to be a new file to s3, even though there is no new nwp data

To Reproduce

Steps to reproduce the behavior:

  1. go to s3 bucket, check timestamp
  2. go to ariflow, kick off nwp-consumer
  3. after finished, check s3 new timestamp
  4. repeat, ECMWF should only have new data twice a day

Expected behavior

Only upload to s3 if there is new data, This could be done by

  1. checking timestamps in the nwp, or.
  2. only if new raw data, then upload. Force upload if latest.zarr doesnt exists

Additional context

https://github.com/openclimatefix/uk-pv-national-gsp-api/issues/337

peterdudfield commented 3 months ago

The deployment is blocked by #156