badger707 / esb-smart-meter-reading-automation

Automated Smart Meter readings data collection from ESB Ireland registred account (regardless of your current supply provider)
https://myaccount.esbnetworks.ie
51 stars 13 forks source link

Downloads entire history #1

Open TheOctonaut opened 1 year ago

TheOctonaut commented 1 year ago

Not really an issue per se but a note (didn't see a way to leave a 'discussion' like in some other projects).

I'm not sure what "startDate" does - it always downloads every record ever. I thought this might be a bug, or that something had changed again, but as it turns out that's the behaviour on the ESBN HDF data page too. It just straight up ignores what you've asked and gives you everything ever.

I only need the first two results for the HA automation I'm making but genuinely can't get ESB to not send me everything. Huh.

badger707 commented 1 year ago

Yeah, I have noticed that too... Well, in my use case I'm ok to get full file every time, and if I need specific part of it I can always transform it in python... I have dropped my initial plan to use Influx + Grafana, does not fit my needs. Instead, I have modified python to act as a sensor with bunch of attributes and dynamic data range parsing by directly pushing billing period dates ranges from HA native dashboard.

image

image

vincentezw commented 11 months ago

Thanks for fixing the script and sharing your experience ❤️

I'd be interested to hear how you've got this configured. I found that new data becomes available only once per day, for the entire previous day. So we're dealing with a chunk of historic data, and without either injecting the data straight into the HA database, or using a history helper module, I couldn't find a nice way to configure this as a sensor.

I have forked the script to export straight to InfluxDB so I can run a daily cron. I made some other improvements too such as a config file, structure and deduplicating before saving to the database. If you'd like to merge those changes I can open a PR.

badger707 commented 11 months ago

@vincentezw nice work on influx integration! My initial approach/testing for influx integration was not as clean as you have it there, every day we learn something new 😄

From my observations, there could be more than few days of refresh delay from EBS, at one point I saw EBS was behind 6 days...

Making this to work as sensor is tricky and requires some weird things to perform on HA installation. My HA is running as VM on ESXi hypervisor cluster (perks of working at virtualisation giant company), I am unsure if it will work/behave same in Docker env, not a fan of Docker - might need some testing but thats not a priority at the moment.

I will need to explore HACS development nuances to wrap this (sensor) in simple package so other folks could install it in less painful way.

PR - yes, lets keep this option open/hot for future revisions.

Ciaran97 commented 6 months ago

Any more progress on the updated guide? specifically looking at getting this data from InfluxDB to a HA sensor.

freddo99 commented 3 days ago

Yeah, I have noticed that too... Well, in my use case I'm ok to get full file every time, and if I need specific part of it I can always transform it in python... I have dropped my initial plan to use Influx + Grafana, does not fit my needs. Instead, I have modified python to act as a sensor with bunch of attributes and dynamic data range parsing by directly pushing billing period dates ranges from HA native dashboard.

  • I'm working on updating instruction with more step by step guide, quick preview:

image

image

This looks great. Thank you for sharing your work. Would it be possible to offer a guide for how to integrate with HA and create the sensors? Thank you.