Adding Stage Changes

[DravenSA]

NOT AN ISSUE< BUT AN IDEA? This might be a long shot, but with all the Schedule updates and the constant changing of stages, is there maybe a way that this integration could display what time the next loadshedding stage will be. eg Stage 1 00:00 - 05:00 Stage 3 05:00 - 16:00 Stage 4 16:00 - 00:00

So we are on stage 3, so the next load shed is like 22:00 - 00:30 but if stage 4 comes into play, that would bring the next Load shedding to 16:00 - 18:30 because stage 3 dosent have the 16:00 - 18:30. Hope that made sense.

If its possible, then the awesome code your wrote for the Schedule display on the front end, that could show the new stage in another colour or bold italic?

Any thoughts on this?

[wernerhp]

With a reliable data source it would be possible, but just like there's no reliable way to get CoCT stages that are sometimes lower than the Eskom stages, there's no reliable way to get what the next stage is going to be. Parsing Tweets or websites requires that the format be consistent or the parser will break. Currently I'm not aware of a reliable source.

[DravenSA]

The m9st rel7able fir me us eskomsepush app. Maybe where they get there source?

Sent from my Galaxy

-------- Original message -------- From: Werner Pieterson @.> Date: 14/07/2022 09:58 (GMT+02:00) To: "wernerhp/ha.integration.load_shedding" @.> Cc: DravenSA @.>, Author @.> Subject: Re: [wernerhp/ha.integration.load_shedding] Adding Stage Changes (Issue #20)

With a reliable data source it would be possible, but just like there's no reliable way to get CoCT stages that are sometimes lower than the Eskom stages, there's no reliable way to get what the next stage is going to be. Parsing Tweets or websites requires that the format be consistent or the parser will break. Currently I'm not aware of a reliable source.

— Reply to this email directly, view it on GitHubhttps://github.com/wernerhp/ha.integration.load_shedding/issues/20#issuecomment-1184123176, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGEIOCQK73WR5LQYMZH5M73VT7CCTANCNFSM53RHN36A. You are receiving this because you authored the thread.Message ID: @.***>

[wernerhp]

They are a business with over 2M customers and push adds to generate an income as well as provide their Business API at upwards of R10k. They won't simply give up their data or sources. I've reached out to them in the past to enquire about their API. A public API may be in the works for the future, but no idea when that will be. They have spent much time (since 2015) and effort to gather and refine the data (very likely in a similar way as I am doing it here), which still breaks sometimes when formats change and then need to be fixed. They likely also have people who can manually update this information in a database based on announcements that Eskom makes.

Looks like City Power might actually have the data in a semi-reliable format. Will need to monitor it. Eskom Twitter Tweet

Screenshot


City Power

Screenshot


Code

```

​Date

​Day

​Start Time

​​End Date

​Stage

```

[fragtion]

Yup they really need to start publishing this information in a more accessible format for developers. We can't all be expected to import these schedules manually like monkeys behind a keyboard. The schedules are inconvenient enough as it is. If they're too incompetent to set up a simple public-facing API, then surely the least they could do is publish them in standard formats like CSV, ICS, or even XLS instead of a bloody PNG image or WORD TABLE exported to html, I mean for pete's sake.

This is a matter of grave national importance affecting automation systems everywhere, and then the only way to obtain the data is through a PNG/JPEG posted on twitter, or a word doc interpreted on a municipal website? I mean what kind of BS is this exactly ?

[wernerhp]

City of Cape Town schedules have a format to them and can be generated, so no need for a CSV or to manually import them. @tiaanv did a python port of another library, which I used for CocT. You can check out the code here.

City of Tshwane seems to have a similar structure, but with some differences. Days 1-31 (CoCT is 1-16 and 17-31). I'm trying to understand this and figured out Stage 1, but every stage is slightly different. Feel free to check out the CoCT lib and create pull requests if you figure it out.

I assume most metros will use similar ways to generate the schedules. Then it just comes down to which area/group your suburb falls in.

Stages is the dynamic component that would be great if they just provided an API instead of posting tweets etc.

[fragtion]

Stages is the dynamic component that would be great if they just provided an API instead of posting tweets etc.

This is what I'm referring to yes. Eskom needs to release these national rolling stages timetables in a more accessible format that is always up-to-date. Stuff like this is too dynamic to interpret manually unless you have someone extremely dedicated or paid to do it. If today is Friday, and it has already been announced that tomorrow will be Stage 1 from 7:00-16:00 and Stage 2 from 16:00-22:00 (for arguments sake), then what does it help seeing all the other stages for tomorrow if they're basically irrelevant? It just adds more noise and possible confusion. Obviously this issue isn't specific to this project, but it's projects just like these (this one being one of the more popular ones of its kind) which happen to depend on such data, and since that data is nowhere to be found in the proper format, ALL of these projects are stalled together for the same reason and we have duplicate issues forming and developers all scrambling to troubleshoot the same problem in parallel. Seems quite inefficient and ridiculous to say the least xD

Given how vital this type of data is for all kinds of automation purposes, and the fact that it is such a huge issue of national importance affecting the lives of every South African, it seems a bit bizarre to me that there's no central data source to point to for obtaining this crucial information. To the point that if Eskom are not going to provide this themselves, someone will need to take up the chore of doing the manual interpretation to complete the dataset basically needed for all of these applications to work properly (assuming they want to offer that much more granularly specific detail in their apps).

[tiaanv]

My two cents worth on the OP. Firstly I don't think the integration should be a visible source of load-shedding schedules etc.... There are many apps/tools out there that can give you that. The critical part is the sensor that assists your system in making decisions. Are we load-shedding? When is my power going to be off next.... I think for the most part this is catered for, barring stage adjustments that take place.

The current stage of load-shedding, correlated to a schedule, to tell you when (on the current stage) you can expect load-shedding is kind of the best we can do for now. With Eskom dynamically changing this on short notice, there is no programmatic way of predicting anything else reliably. I would love nothing more than for this to be predictable, but I doubt it will happen. on top of that Local municipalities have also added an additional layer of unpredictability.

Having a more accurate COCT "current stage" would be a step up. The COCT have an app, that I have tried to reverse engineer, but it has nothing in it, it's just HTML that's packaged. It in turn does some calls to an API to get the COCT stage, but that's using SSL (expected)... try as I might, I could not get inside the SSL, even with a MITM attach, since they seem to use pre-seeded certificates... So I can't see the URL or packet contents, unfortunately..

Long and short is that at least right now, there is no reliable way to get to the real stage, not even for "right NOW", let alone for in a few hours. even if you could it's a pretty fluid situation, and probably would not be reliable enough.

My practical experience is the following. I use my load-shedding status to get my house in shape an hour before load-shedding kicks in.. I have a solar PV system and batteries, so ensuring the batteries are topped up is key. The rest of the actions I take when the power actually goes off, rather than when the load-shedding schedule says the power will go off.. If I have topped my batteries up unnecessarily, then so be it... I ignore the COCT lower stage.

I would be interested to hear your individual use cases...

[wernerhp]

Added the stage forecast for the next few days. It's very experimental, but you can Redownload and install master via HACS

[fragtion]

Hi @wernerhp

So awesome that we now have the option to include the Stage forecast. Amazing work!

In my opinion however, we need a third type of widget... Basically a "schedule" which takes the "stage forecast" into account, which will then provide the simplest and most succinct schedule to the user of what they are most likely to expect and when. This should be implemented on the library level, so that it can also be used for automation purposes eg for industrial, medical equipment etc (in theory - obviously reliability is not guaranteed etc etc... this is all best effort and should probably be stated as such in the README, too btw).

For example, a solar system could be controlled by home assistant to start charging the batteries from mains x hours before the next load shedding hits home, and this logic would only be reliable if such a schedule existed, which right now we don't have yet because the user still needs to interpret the schedule for stage X against the Stage forecast to determine what will precisely happen in the future

I've been dabbling around a bit with your example.py script and noticed that with Eskom as the provider, it's only possible to request a schedule for a single stage at a time. This isn't the case with City Power, where the full timetable (schedule for all stages) can be returned at once if you don't include a stage in the URL request (directly via browser). I'm sure there are other providers where this limitation could also present itself.

So I think the safest solution is to create new functions, ie forecast_schedule_handler and forecast_search_handler which then basically calculates a "max stage" from get_stage_forecast(), and then requests separate schedules from provider for each stage between 1..max_stage (or simply just the sanitized list of stages returned by get_stage_forecast), and then finally compares the data from all the calendars against the stage_forecast schedule (which stages at what times) to produce a precise forecast for an area, rather than being limited to only the stage forecast itself, and a schedule for only the current stage

I'm making some progress and will be happy to submit a merge request if this is a feature you'd be interested in but don't have the time to develop. However, my python abilities are nowhere near your level yet, so I think first prize would be if you could expand the library to do this yourself :) Your sentiments on this?

[fragtion]

OK I hacked something together at https://gitlab.com/fragtion/load-shedding/-/commit/8562122f792474d1c324101a7f9bbf50002a96d1 Function of interest is forecast_search_handler which is called by @app.route('/forecast_schedule'). Still need to adapt forecast_schedule_handler

Not sure if you would want to implement this natively within the library itself as opposed to a script like this? Seems like a must-have for the home assistant plugin at least, IMO

[wernerhp]

@fragtion

In my opinion however, we need a third type of widget... Basically a "schedule" which takes the "stage forecast" into account, which will then provide the simplest and most succinct schedule to the user of what they are most likely to expect and when.

We'll get there. This was just a quick proof of concept to see if it holds up. The data is scraped from City Power's website, so there's no guarantee that it will work in the long term. There's no point in building elaborate things if we don't have reliable data. I'll probably iterate on this, when I have time, to get only the relevant schedules for the area.

I've been dabbling around a bit with your example.py script and noticed that with Eskom as the provider, it's only possible to request a schedule for a single stage at a time. This isn't the case with City Power, where the full timetable (schedule for all stages) can be returned at once if you don't include a stage in the URL request (directly via browser). I'm sure there are other providers where this limitation could also present itself.

The stage forecast is part of the citypower lib because the data is scraped form City Power's website. If you call the libs directly then you'll only get data that's available for them.
The load_shedding module abstracts things away from providers. Since we can only get this data from city power, it gets called internally irrespective of the provider you pass in. The readme on the dev branch is more current than master.

[tinuva]

Having a more accurate COCT "current stage" would be a step up. The COCT have an app, that I have tried to reverse engineer, but it has nothing in it, it's just HTML that's packaged. It in turn does some calls to an API to get the COCT stage, but that's using SSL (expected)... try as I might, I could not get inside the SSL, even with a MITM attach, since they seem to use pre-seeded certificates... So I can't see the URL or packet contents, unfortunately..

No need to reverse engineer the app. You only need to grab the logs (it logs A LOT) and I was able to find the endpoint. You can see it in the works on https://github.com/tinuva/ha-coct-loadshedding which is mostly based on your and sweartjean's HA integrations.

That said, the ESP public API for private individuals are in beta testing now and Werner does have the details. I think ESP will have the most reliable information, and includes the same future stages that the ESP app shows.

[tiaanv]

Having a more accurate COCT "current stage" would be a step up. The COCT have an app, that I have tried to reverse engineer, but it has nothing in it, it's just HTML that's packaged. It in turn does some calls to an API to get the COCT stage, but that's using SSL (expected)... try as I might, I could not get inside the SSL, even with a MITM attach, since they seem to use pre-seeded certificates... So I can't see the URL or packet contents, unfortunately..

No need to reverse engineer the app. You only need to grab the logs (it logs A LOT) and I was able to find the endpoint. You can see it in the works on https://github.com/tinuva/ha-coct-loadshedding which is mostly based on your and sweartjean's HA integrations.

That said, the ESP public API for private individuals are in beta testing now and Werner does have the details. I think ESP will have the most reliable information, and includes the same future stages that the ESP app shows.

Nice Work!!! Will check it out.... Love it when I'm the dumbest guy in the room!

[wernerhp]

v0.6.0-dev (Pre-release) - Feedback