aimlac-7564616D / crawlers

Data crawlers and parsers for various energy APIs.
0 stars 1 forks source link

Elexon B0620 & B1620 #15

Closed fjebaker closed 2 years ago

fjebaker commented 2 years ago

Added flows for scraping

In the SLIMJaB code, the B1620 data is never used as is, and is always aggregated. Infact, the incoming data for B1620 is pretty degenerate, so I put the aggregation functions directly into the flow, and then we store the total quantity in a given period for every resource, per row in the database.

Screenshot

Screenshot 2022-03-19 at 14 53 28

Note the flows are starting to get a little messy, so I think we should open an issue about tidying this into sub-flows, so we can abstract common reused sections.

fjebaker commented 2 years ago

Note these new tables also cylce on the 24 hour period.