White flashes every 5 minutes - seems related to HA integration

harueg commented 2 years ago

What happened?

Leds flash white every about 5 minutes, hard to tell the exact time in between.

To Reproduce Bug

Setup with esp32E 4MB on AZ-delivery board. 8 outputs with 144 leds each. GPIO 2, 13, 14, 16, 25, 26, 27, 32 Each output is a segment. Each segment is running different builtin effects.

Expected Behavior

Obviously no flashes

Install Method

Binary from WLED.me

What version of WLED?

WLED 0.13.0-b6 (build 2112080)

Which microcontroller/board are you seeing the problem on?

ESP32

Relevant log/trace output

How would I log that ?

Anything else?

Thank you

Code of Conduct

[X] I agree to follow this project's Code of Conduct

blazoncek commented 2 years ago

White flashes (across all leds) usually mean there was an interrupt while updating leds. Try reducing number of outputs to see where flashes start occurring. If this is with only 1 output it may mean your ESP is having other work to do during led updates (wifi broadcasts, usermods, etc.) preventing it to properly update leds.

harueg commented 2 years ago

Hi blazoncek Thanks for the answer ! Is there a way to estimate the max number of leds and the max number of outputs for a certain setup. There is usually a 20ms pause between sending the led data. Wouldn't it possible to put the other "work" in this timeframe ?

blazoncek commented 2 years ago

interrupts occur at any time

harueg commented 2 years ago

Yep, I guess that's the idea of an interrupt.

harueg commented 2 years ago

so from your experience what would be the maximum configuration without having flashes ? Setup either with ethernet ddp, data from PC or playing fseq files from sd card.

blazoncek commented 2 years ago

I am just explaining my experience. I would suspect a network traffic (mqtt, udp or some other polling your ESP) may be the cause but it may also not be. It may be other hardware in your set-up (sensors and such). Another possibility is a faulty strip or noise on the data line. You will need to diagnose your setup by compiling source yourself by adding debug output to various parts of the code. Still I am not sure this will help narrow down the problem.

ghpk commented 2 years ago

Hi, Are you sure you have sufficient Amps to run 144x8 Setup ? from your post I am assuming you are using WS2812b with 144Leds/Meter. Can you please try injecting Power from other Side of each strip and see if that helps. How long is the Data Line wire from ESP32 Output pins ?

Try running the same setup with 144x4 LEDs and see if that helps.

thepilchers commented 2 years ago

I was noticing a similar issue - an occasional inconsistent flash or "hiccup" while running my LED matrix.
Configuration: Dig-Quad (EPS32) with ethernet running 17 meters of 30 pixels/meter WS2811 in a matrix configuration (total 510 pixels) on 1 channel (pin). 5V 30A power supply set at 5.2V, power injection every 2 meters. Approx 3 meter, 18 gauge, 3 wire cable connecting Dig Quad to matrix. I tried the following to correct:

substituting 1 meter pixels at a time with a new pixel strip
Adding a 2nd channel (pin) at 8m
Introducing data boosters at each end of the signal wire (2 different types, Intermit.Tech and Holiday Coro)
Using a different 5v 8A power supply
Using a different output pin on Dig Quad
Varying brightness

None of these made a noticeable difference. I then reverted WLED to v0.13.0-b5 (build 2111170) which fixed the problem.

blazoncek commented 2 years ago

None of these made a noticeable difference. I then reverted WLED to v0.13.0-b5 (build 2111170) which fixed the problem.

I don't think it did, it just made it latent. 🤷‍♂️ As I tried to explain above, if you see white flash across all LEDs it will most often mean signal timing issues (caused by interrupts or interfering code) and it may reappear. Of course there is a possibility that WLED code is sub-optimal in places and optimizations in future will help.

thepilchers commented 2 years ago

You are absolutely correct - I left it running (color 1: RED, color 2 & 3:BLACK, Android effect) with b5 for about an hour and the "hiccups" are occurring again. Possibly interrupts from the ethernet interface on the Dig Quad?

blazoncek commented 2 years ago

Interrupts occur all of the time. Usually they are quickly processed including ethernet. If they occur during LED update (which itself may need interrupts for timing) this timings may get skewed hence flash. You may need to monitor network traffic using WireShark or something similar to see if network packets coincide with flashes. If they do it may point to a sub-optimal code in WLED (what exactly will depend on the packet). If they don't there may be other things that will interfere (I noticed that a poorly written usermod was a culprit of such flashes once).

softhack007 commented 2 years ago

I also had white flashes on my setup (ESP32, 300LEDs SK6812, 15Amp power supply). I found that they were largely caused by electrical problems:

5V level shifter (TXS0102) was about to die, and stopped working the next day. Had to replace it.
LED data cable was thin, long, and not shielded - so it acted like an antenna. Using a shorter cable reduced flashes and colour errors.
I had the first power injection after 50 LEDs, however I didn't connect 5v and gnd lines at the beginning of the strip because I was afraid to toast my thin dupont wires... 💡 as it turned out, a full "power connection" at the first LED reduced flashing considerably.
reducing the distances between power supply, controller, and LED strip helped - basically I nailed my ESP32 to the top of a wall, to be as close to the strip as possible :stuck_out_tongue_winking_eye:

.

If they occur during LED update (which itself may need interrupts for timing) this timings may get skewed hence flash.

Reminds me of something - Could there be some interferences with asynchronous wifi events during LED updates, maybe from AsyncWebServer? NEOPixelBus suggests a workaround for ESP32: https://github.com/Makuna/NeoPixelBus/wiki/ESP32-and-RTOS-Tasks

Is WLED implementing something similar already?

howardtopher commented 2 years ago

I know I'm a few months late commenting on this one, but I've noticed a similar issue recently. I run my lights from sunset to sunrise every night, but just on solid white. I ran the built in "flow" effect with red, white, and blue for July 2-4. On the 3rd, I noticed a white flash happening randomly. Sometimes 30 seconds apart, sometimes a couple minutes apart. Tonight (the 4th), I sat outside and stared a the lights for a good 10 minutes with no flashes at all.

Christmas was the last holiday I ran a pattern other than solid white. The Christmas pattern was run for more than a month (thanksgiving to new years), with never noticing any flashes. In fact, I've never noticed any flashing in the year I've run WLED in this install until last night. In the time between Christmas and July 4th, I've done two things - 1) update from 0.12.0 to 0.13.1, and 2) start running Home Assistant and added the WLED into there. The solid white still runs perfectly.

At first I figured it was the update that did it assuming new code had a bug or something. However, I did a packet capture tonight and noticed that Home Assistant polls the WLED web ui for two different URIs (/json/si and /presets.json) and it polls both of those every 5 seconds constantly. Now I'm thinking that Home Assistant is just polling it too often and getting in the way of signal generation. Prior to Home Assistant I ran Hubitat which doesn't have an integration so all I did to control it was use the json API (no continuous polling). I'm considering deleting the Home Assistant integration and going back to using the json API directly and see if that helps.

My install is a Dig-Quad (esp32) with about 22 meters of 60 LED/meter WS2815 (12v). Running 2 separate data channels with one about 700 LEDs and the other about 600 LEDs, just over 1300 LEDs total. The flashes happen on both data channels at the same time, which would indicate a signal generation interruption.

blazoncek commented 2 years ago

Reminds me of something - Could there be some interferences with asynchronous wifi events during LED updates, maybe from AsyncWebServer? NEOPixelBus suggests a workaround for ESP32:

https://github.com/Makuna/NeoPixelBus/wiki/ESP32-and-RTOS-Tasks

Is WLED implementing something similar already?

Sorry to see this post so late. Thanks for the insight!

Indeed that could prevent "some" of the flickers related to web UI access or web API calls. Still these are in minority and only happen when you interact with web interface (UDP has a simpler code and may be less prone).

Nevertheless it may be good to implement seggested approach at some point.

thepilchers commented 2 years ago

I too am running Home Assistant with WLED. Based on howardtopher research above I removed the WLED controller from HA that had the issue. After rebooting HA (and a few hours of testing) the problem seems to have gone away. At least in my case it appears the HA polling was the issue, thanks, howardtopher, for your research and comment!

EmmanuelVella commented 1 year ago

I also had flashes during 1 second every few minutes (often on low white colors). Tried everything, then I disabled the Home Assistant integration : no more flashes !

ballakers commented 1 year ago

I think this has something to do with HA looking for the WLED firmware updates?!?! Also noticed it sometimes when opening or closing the WLED app on my phone (but that may have been from messing with LED preferences). A reboot after messing with any LED preferences for sure helps and make sure to have the right preset for your segment(s). Going to remove the light from HA for now too as mentioned above. The firmware unavailable in HA logbook happens on all my lights it seems but only get flashes on my larger setup it seems. Still not sure if it's a mix of hardware and software issues or just one or the other hahah - I guess we solder up another controller and plug that one in haha.

SK360 commented 1 year ago

Having this issue on multiple WLED controllers, both wifi and ethernet and like others commented it seems to be directly related to HomeAssistant.

I can't remove the lights from HomeAssistant as it defeats the entire purpose

howardtopher commented 1 year ago

Having this issue on multiple WLED controllers, both wifi and ethernet and like others commented it seems to be directly related to HomeAssistant.

I can't remove the lights from HomeAssistant as it defeats the entire purpose

You can still use WLED in Home Assistant without using the built in integration. I've since disabled the integration and switched to using REST API calls to control WLED. It's a bit more complicated, but removes the constant polling that HA does to WLED and therefore stops the white flashes.

SK360 commented 1 year ago

Having this issue on multiple WLED controllers, both wifi and ethernet and like others commented it seems to be directly related to HomeAssistant. I can't remove the lights from HomeAssistant as it defeats the entire purpose

You can still use WLED in Home Assistant without using the built in integration. I've since disabled the integration and switched to using REST API calls to control WLED. It's a bit more complicated, but removes the constant polling that HA does to WLED and therefore stops the white flashes.

That’s a good idea actually. Haven’t setup any rest calls from HA but doesn’t look hard

The only downside I can see is I use a lot of Scenes for setting lighting all around the house and I’m guessing I couldn’t use WLED with those scenes anymore

mattbruman commented 1 year ago

Is there a bug report with HA yet? Maybe we need to have them add a way to adjust the polling time. Maybe have them only poll the light when actually viewing the entity on a dashboard. And be able to disable polling unless you are making changes and triggering automations.

algovera commented 1 year ago

I have the same issue, my environment has 3 WLED setup, with 450, 1300, and 1500 LEDs respectively.

My 1505 LED setup has been flickering and I tried every methods possible to resolve the issue.

I spent days solving the issue, and confirmed it is HA causing it.

Finally I replaced the ESP32 and the new chip worked flawlessly, as soon as I joined it to HA and the problem comes back

adammartin1981 commented 1 year ago

I've got a similar issue.

My lights don't flash white - just change colour - but I've only noticed it on a solid colour vs an effect.

When I shut down the Pi (running HA) the lights stop flashing/flickering.

On my HA I have a number of things running, but the bits that I want to look at (which I hunch might be causing it) are MQTTBroker (Mosquito I think) and the WLED addon.

Plan is to disable one at a time and see if either of those makes a difference. (WLED for constant searching and polling of devices, to query state etc and MQTT just because it's there merrily broadcasting and listening)

On the WLED device side of things, I've already disabled receiving UDP packets and trying to no emulate various devices incase for some reason the HA was pinging them at various intervals, but that didn't work.

I was also running version 13 and just updated to 14 (WLED) to see if that made a difference (it didn't).

I have changed my MQTT channel from 11 to 15 (not sure why it would, but changed anyway) and that didn't make a difference.

I'm running 980 leds on my set up - but lowered the amount being addressed (incase of the ~800 limit mentioned on the config) to 500 but it still flickered.

I dropped the brightness too (incase it was a PSU issue) but that didn't affect it either.

Happy to be a guinea pig if anyone else has some ideas/thoughts to try.

adammartin1981 commented 1 year ago

Could it be related to this? https://kno.wled.ge/features/presets/#a-note-on-performance (and HA doing something with it).

I'm running a few presets and have them listed inside HA.

It's a long shot - but the WLED api doesn't provide the preset information (names etc) so you have to muck about with the presets.json file in order to read the list etc..?

I know that the presets are listed within HA.

A good/better way of seeing the issue it is with a smooth effect that transitions between colours slowly - for me - the effect became even more profound as it was clearly being interrupted for different periods of time.

I'll see if I can read the WLED HA integration code at some point and work out what it's doing

I've just had to disable (and then restart) the WLED integration for it now to no longer flicker so I'm keen to get it back without the flicker 😄

blazoncek commented 1 year ago

Yes, HA polls WLED too often and causes resource depletion. A flash in such case means ESP was not able to update signal on the wire in a timely fashion. LEDs then interpret data randomly. presets.json is served as whole to HA and HA does the parsing. If presets.json is large it will cause bigger resource depletion if served too often.

Even though it makes sense to use WLED with HA (in some configurations), WLED never was an extension of HA. There are just too many things you can do with WLED that cannot be comprehended with HA integration.

swampylee commented 1 year ago

Interesting thread. I had exactly the same issue with a 32x32 matrix display (4 x 16 by 16). As soon as I disabled the HA integration the random flashes stopped. How do we get this looked at or is the only option to disable the integration in HA and move to using REST calls?

msfuture commented 1 year ago

Same here. TM1814 strip working finde using WLED alone. As soon as the Home Assistant integration accesses the WLED instance it begins flashing (red in my case, as the TM1814 falls back into a demo mode when no data is received). Really annoying. But I think we need a issue report in HA...

harueg commented 1 year ago

what is the default state of HA integration ? active or inactive ?

dmonty2 commented 6 months ago

Could the WED project implement xTaskCreatePinnedToCore() code to utilize the ESP32 2 cores?

to handle all API network requests
to run the led strip

Would this fix the issue?

Here is an overview of pinned to core: https://randomnerdtutorials.com/esp32-dual-core-arduino-ide/

There is already one usermod that runs a display on a 2nd core: usermod_v2_four_line_display_ALT.h

softhack007 commented 6 months ago

Hi,

Could the WED project implement xTaskCreatePinnedToCore() code to utilize the ESP32 2 cores? to handle all API network requests to run the led strip

We are already utilizing several tasks, and network requests (Webserver, JSON API, websocket) are already running in their own tasks "async_tcp" at higher priority. I think the main problem with HA is that request are shot towards WLED at an insane rate, and serving JSON files from filesystem also has a timing penalty, that might lead to flashing.

In fact there is a recommendation by makuna (for NeoPixelBus) to run LED updates in a separate high priority task that communicates via queues. However this is not easy to implement into WLED. I've tried a simple variant of it some time ago, and it solved nothing 🤷 .

Here is an overview of pinned to core

Thanks, I know you just want to help. I think we are already much more advanced than what is described on randomnerdstutorial.

Would this fix the issue?

See above. Its more complicated, and forking off a new task would not solve problems by itself, as both cores are already well-utilized on esp32. What could help is to revise our task architecture, and define real time task priorities based on task deadlines - basically just what's done for most satellites flying around earth ;-). But its a lot of effort to do such a "response time analysis", and would not help on 8266 as that device does not have tasks.

Actually there is a new release of NeoPixelBus (v2.7.8) with reduced jitter - it might be worth to check it out. https://github.com/Makuna/NeoPixelBus/issues/751

blazoncek commented 6 months ago

@softhack007 It may be good to revise new I2S driver which has the ability to drive up to 16 outputs at the expense of RAM.

dmonty2 commented 5 months ago

Some have mentioned that switching from the WLED Integration interface to a MQTT interface causes the white flashing to go away. Here are some experiments with MQTT and my results.
https://github.com/dmonty2/homeassistant_wled_mqtt/

github-actions[bot] commented 1 month ago

Hey! This issue has been open for quite some time without any new comments now. It will be closed automatically in a week if no further activity occurs. Thank you for using WLED! ✨

jsalbre commented 1 month ago

Anyone had any luck finding a solution to this?

dmonty2 commented 1 month ago

@jsalbre - I found disabling "automatic brightness limiter" under "LED Preferences" helped reduce flashing. I just had to ensure I had the right power supply instead of using a software hack to limit the power.

harueg commented 1 month ago

I must admit I didn't check it with newer releases. And all my spare 144px/m strips have been used up for repairs. I have some setups with 276 rgbw pixels per gpio and working perfectly.

Aircoookie / WLED