Optimize Speed

[theaceofthespade]

First of all, I just want to say this project is absolutely brilliant!

I'm experimenting with some more dynamic setups (eg, position based light changes, chaser effects, twinkling, etc). Right now I'm just running everything from rest calls directly from the browser (also as a note to that, I adjusted the api code to respond to OPTIONS requests and added cors support, so I can find a way to share that if anyone is interested).

Does anyone know the best ways to optimize/speed things up? I assume the HTTP layer is adding a lot of overhead currently, and I get about 300ms response times on average, so I assume that using a different protocol I could cut that down a bit. I've got a ton of ideas for speeding things up, from grabbing a stronger board all the way through writing something to find single-use-groupings to transmit simultaneous updates faster (eg grouping four otherwise unrelated lights so that one call can change all their brightness in one situation), but figured it would be good to drop in here and see what everyone thought would be a good starting point.

[theaceofthespade]

So for my first experiment I just built two hubs, and alternated sending http requests between them. This worked great, so the next thing I want to look at is simultaneous broadcasts from the same unit. Of all the areas involved in this project, rf stuff is my weakest subject by far, so I'm looking at either finding a way to broadcast more than one command at a time from the rf unit, OR having two broadcast units on the same ESP (or both honestly), but I need to do more research there. Either way, it looks like I'll have to write some sort of broadcast queue so that the http server and the broadcasts can be more concurrent.

Any input (or warnings) are welcome!

[sidoh]

Hello,

HTTP is probably the slowest integration because a connection has to be established each time. The best option currently is MQTT, which has the same functionality and analogous structure (topics <-> routes, and payloads are the same).

There already is a packet queue actually, but this doesn't help with throughput. There, the vast majority of the bottleneck is the nRF24. The radio code has been optimized a fair amount, but I'm sure there's room for improvement. Sending packets through multiple nRFs is possible in theory, but my guess is it'd be difficult to get the code working.

This page on the wiki covers some things that might help you here (especially the last two sections):

https://github.com/sidoh/esp8266_milight_hub/wiki/Performance-Tuning

[theaceofthespade]

Thanks so much or the reply - that guide was extraordinarily helpful! Firstly, let me just say that I'm posting here to contribute, but if this format is tiresome or not the right place for this, let me know!

I definitely have a couple challenges going for me. For one, after much experimentation, I finally figured out that the shape of my ceilings (slanted and angular) make radio broadcasting all over the room a real challenge. I live in a loft apartment, so all the ceilings and a lot of the walls are slanted around 30 degree angles. Additionally, there are old, thick 2x4 beams with god knows how many layers of insulating paint which seem to provide a real challenge to penetration.

Right now, I'm using 2 different receivers to change the lights. I've modified the rf units with external antennas following these tutorials (1 of each). Both were definite improvements, but I found that the signal was WAAYYYY better by coiling the actual antennas. https://www.instructables.com/id/Enhanced-NRF24L01/ https://www.youtube.com/watch?v=NpMnauHeR7Y&t=37s

Even with two receivers and antennas, I need a lot of message redundancy on at least mid and high in order to get the lights to change reliably. In all tests, sending too many instructions to the broadcasters without adequate time to broadcast has caused severe failures. Despite finding the mqtt implementation to be faster, I'm sticking with http for now, as I can't seem to send mqtt broadcasts out faster than 500ms each without signals being missed for the lights. With http, I can wait until each one comes back and immediately send the next change, and that actually winds up being faster on average, so it looks like http is not the bottleneck I thought it was (barring any sort of concurrency that may be in there).

On the bright side, when I had it broadcasting on one channel, with no listening, and the throttling enabled, it was only really reliable about 50% of the time, but during that 50% it was able to switch the lights exactly as fast as I was wanting - fast enough to achieve believable effects like having a "virtual color ball" move around the room and update the lights accordingly.

CONTRIBUTING So as far as contributing to the project, I'm looking at coding up a couple of things (aside from my speed-related obsessions), and I'd like input on the their desirability. For one, I'm looking at moving the current web interface to a slightly different route (maybe /default, or /admin, or something like that), and allowing someone to upload a web bundle the same way they currently can upload new firmware. That way people will be able to upload a little page/various web assets to be presented from the hub url without altering the source code, and we can have multiple, different ones floating around to adapt the system to different purposes. I've also already coded up cors support for all routes if that's something that anyone would be interested in.

NEXT STEPS As far as next steps, I'm looking at moving in a few directions (though I'm super open to any additional input or direction). For one, I'm going to try a power amplified radio module to see if I can just brute force the issue that way (as well as change the neighbors lights I suppose). I'm also going to be doing a bit of a deep-dive into antenna shape, as well as experimenting with some of the low-level broadcast queueing and concurrency concepts.

[sidoh]

Sounds like you're making progress! Glad to hear it.

For one, I'm looking at moving the current web interface to a slightly different route (maybe /default, or /admin, or something like that), and allowing someone to upload a web bundle the same way they currently can upload new firmware.

I'm hesitant to move the default UI from where it's at. I'm not opposed to making it possible to side-load an alternative UI at a different route, but I think I'd want to explore alternatives.

Speaking of alternatives, you mentioned one that makes a lot of sense to me:

I've also already coded up cors support for all routes if that's something that anyone would be interested in.

If the entire API is CORS-enabled, you can just host whatever UI you want pointing at the hub, right? I like this a lot more because it's less liability (aka code :)) to maintain.

CORS support sounds like a great addition. In a pretty funny coincidence, another user was asking about CORS support in #561. :)

As far as next steps, I'm looking at moving in a few directions (though I'm super open to any additional input or direction). For one, I'm going to try a power amplified radio module to see if I can just brute force the issue that way (as well as change the neighbors lights I suppose). I'm also going to be doing a bit of a deep-dive into antenna shape, as well as experimenting with some of the low-level broadcast queueing and concurrency concepts.

All sounds like interesting stuff.

I recall that @khmann worked on some super cool looking high-throughput milight stuff a few years back (see this project on their github). Personally have no further context on it other than the impression that it looks super cool.

(khmann actually made the first significant pass at optimizing the nRF handler code this project uses).

[theaceofthespade]

Wow, KHMann's stuff got me pretty excited! I'm diving right in on some of that!

As far as the web interface, I agree, and that was poorly phrased on my part - what I meant was that there would be two places where you could find the default UI, such as "/" and "/admin" or something like that. Then from the default UI, you could sideload a new UI, which would replace the one on "/" but not the one on "/admin." The way I was looking at doing it was having it be only a single html file that one could upload, so we wouldn't have to mess with any zipping/unzipping/image serving, and as you pointed out, there's a lot less code to maintain then too.

[sidoh]

Yeah, I'm following (the sentence "...hesitant to move..." in my last message was extraneous).

I think I'd like to get a general sense that people would use this before adopting support for side-loaded UIs. My intuition is that it'd add more code and constraints on how SPIFFS is used than it's worth if there isn't more general interest.

I think if we've got CORS, you can host the application pack somewhere else, right?

The way I was looking at doing it was having it be only a single html file that one could upload, so we wouldn't have to mess with any zipping/unzipping/image serving, and as you pointed out, there's a lot less code to maintain then too.

Sounds good. In case it's helpful - this is roughly how the existing web app works, except it's stored in progmem instead of on SPIFFs.