search

beyarkay / eskom-calendar

Get your loadshedding schedule in your calendar and never be left in the dark! Open-source, up-to-date, and developer friendly.
https://eskomcalendar.co.za
GNU General Public License v3.0
190 stars 35 forks source link

Issue: machine_friendly.csv too large for embedded solutions #341

Open kebasaa opened 1 year ago

kebasaa commented 1 year ago

I am trying to use either the _machinefriendly.csv, or one of the ics files to read and process on an ESP8266. However, neither can be loaded directly from Github because the libraries don't support redirects, and the TLS encoding (https) is hard to implement. The pastebin link would work (when using simple http), but the file is too large to load on such a small embedded device. I have also tried to load the ics file directly from Github, but again the redirects and https don't work.

Could you create a smaller dpaste that is smaller? I could think of a few different possible solutions:

Would this be possible? Thanks

beyarkay commented 1 year ago

Hey!

Thanks for raising the issue, I've used ESP8266's before and it's great seeing someone try integrate them with eskom-calendar. I've very keen to try make something work, because it would be great to get loadshedding schedules on embedded devices.

I could definitely modify the build scripts to upload several smaller files to dpaste, but I've reached out to the maintainer first to check that I'm not about to leave them with a $1000 AWS bill.

Having the files on dpaste might not be an ideal solution for you though, because the URL is not constant. It changes every time the file is updated (that's a limitation of dpaste).

Eskom-calendar can get around this because I write the URL to the Release notes (see the link right at the end, If you encounter CORS issues, you'll need to use this [pastebin link](https://dpaste.org/geD8J/raw) instead), but that won't be a solution for you, since you can't afford to download multiple kilobytes of release notes each time.

Having a reduced pastebin file might work, but it still wouldn't make it easy for you to figure out what the link actually is.

Long term I want to move to a dedicated server, and then we don't have to deal with all this github trickery and I could just straight up give you an API to work with.

But for now, do you have any idea what the maximum file size the ESP could handle? Like I said, I absolutely want to make eskom-calendar friendly for the ESP8266, but it'll take a little bit of work to get there.

DarrenOfficial commented 1 year ago

I'm not about to leave them with a $1000 AWS bill.

That shouldn't happen, as I don't use AWS…

Long term I want to move to a dedicated server, and then we don't have to deal with all this github trickery and I could just straight up give you an API to work with.

Could you reach out to me via discord DarrenOfficial#3451, I'd be more than happy to share dpaste resources. It'd be better for me and you to save on time, resources & cost, as you can update the file without the URL changing. Solves this issue:

Having the files on dpaste might not be an ideal solution for you though, because the URL is not constant. It changes every time the file is updated (that's a limitation of dpaste).

Love your project btw ❤️

kebasaa commented 1 year ago

Thanks for that!

But for now, do you have any idea what the maximum file size the ESP could handle?

Well, unfortunately I don't know exactly, as it varies with the arduino sketch. I'm trying to use the H4 library's ArmadilloHTTP because it allows for asynchronous handling. ArmadilloHTTP only specifies that it allows for data up to about half of the free heap to be sent / received. In my case, I had to activate TLS (which uses a lot of space, so half the remaining heap corresponds to approximately 20000 bytes.

Note that the main developer of H4 passed away last year, but some people have been keeping the library going. I've updated the H4 installer to work with those libraries.

kebasaa commented 1 year ago

Hi, I just wanted to ask if there has been any progress on this issue? I realise that it might be a bit complicated... Thanks

beyarkay commented 1 year ago

I've been wrapped up trying to complete my master's degree I'm afraid, and don't really want to implement a half-arsed solution since it's likely to come back to bite me ):

Can I ask which calendar you'd be interested in specifically? If it's just one or two, I might be able to knock something together quickly for you, which should work until I can actually write the API for a proper solution.

kebasaa commented 1 year ago

Of course, the master thesis is the priority, no question! I'm about to start a postdoc at Stellenbosch University myself

Something quick would be great... I'm interested in the City of Cape Town Areas 9 (Pinelands) and 15 (Observatory)

beyarkay commented 1 year ago

Good luck for the postdoc! ba07592083f6467a39850f039056db337a4c2061 should have a temporary fix for this issue.

I've added some code to write out cpt-9 and cpt-15 as their own CSV files, which will be available (and continuously updated) in the same way as machine_friendly.csv. I'll post a link once the Build job has actually uploaded them to the Release.

They won't be available on dpaste by default, although that should be possible (if a bit of a pain) to do if it will be useful to you. Just let me know 👍 .

beyarkay commented 1 year ago

😋 looks like it's all good! Let me know if you run into trouble, but hopefully <2kb is small enough?

kebasaa commented 1 year ago

Yes, this is perfect, thank you. The only issue now is that I can't parse redirect links and HTTPS easily from the ESP8266, so do you think a stable dpaste link (that doesn't change) would be possible, as discussed above? Thanks

beyarkay commented 1 year ago

Hmm, okay let me see what I can do

kebasaa commented 1 year ago

Hi @beyarkay Just wondering if you managed to finish the thesis work? Do you think it'll be possible to get an dpaste link that doesn't change for the different areas? Thanks