Bug report

Problem

When I set the effect interval from a key on my remote control, the interval is set. After it writes the value to json storage it then starts a loop where the json buffer size continues to increase until the board is power cycled. This does not happen when other events trigger json to store

What I see in the console:

(W) (NotifyJSONWriterThread)(C1) >> Notifying JSON Writer Thread (W) (NotifyJSONWriterThread)(C1) >> Notifying JSON Writer Thread (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 6144 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 8192 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 10240 bytes

Steps

Have the debugger console running in VS Code.
Somewhere in code trigger: effectManager.SetInterval(30000);
Watch the json buffer size increase by 2048 bytes (as defined in jsonbase.h).
Watch the effects being drawn on the leds lag more and more.
Power cycle the board and the effect interval now reflects the value triggered by the remote. The buffer increase has stopped.
Press the button that changes the interval. Watch the console fill up with SerializeWithBufferSize notices.

Example

effectManager.SetInterval(30000);

Notes

Until today, I was running on an outdated codebase (last changes from main branch merged into the branch I'm working on are from October 9). I merged the changes to the main branch from the pull requests dating from October 17th onward. I don't think the error showed up until I did that.

I suspect there are other instances where this error is popping up, but I am having a hard time isolating when / where. I just pushed a fresh build after erasing a device and it is giving the out of memory notice as well. But that appears to be related to a couple of audio effects I added to effects.cpp. If I disable those and disable audio, the out of memory warning goes away.

I will verify on a different board later and see if the error is present there as well.

This happens when a SerializeToJson() function called by SerializeWithBufferSize() returns false - which it should only do if it actually does run out of memory while serializing. Can you provide a link to the branch you're working on, and tell me what PlatformIO env you are building?

I just tested this with the current version of the codebase on Mesmerizer and Spectrum (the latter running on an M5StickC Plus) by changing the effect interval using the web UI (which calls the same g_ptrSystem->EffectManager().SetInterval() function in the end) and the problem does not occur on either.

What I find interesting in your logging is the double NotifyJSONWriterThread trigger - only one of those should really be originating from the SetInterval() invocation.

@rbergen https://github.com/revision29/NightDriverStrip/tree/remoterework The environment is espcross. it's based off of the basic demo environment.

I had to put the code back in a broken state. Having you look at the branch is like inviting a stranger over to a messy house. So, don't just judge the mess. 😃

Anyway, I don't know if the problem is caused by me not having everything properly setup for audio compatibility, if there is something amiss with the web server, or maybe a combo of both.

@rbergen I clicked the wrong button when responding and accidentally closed the issue. So I reopened it.

NotifyJSONWriterThread is being triggered twice. On the remote I am using the AUTO button to either set the interval to 30 or 0. If it is setting to 30, I go ahead and trigger the next effect to give visual feedback that the button press worked. So, it writes the setting for current effect and then writes a second time for the change in interval.

Now that I know what methods are involved, I can dig a little deeper.

I changed 4 things and then discovered the problem: Enabled wifi and web server on the device. Enabled audio. Added two audio based effects to effects.cpp. Merged the latest changes from the main branch.

I will see which combo of those changes result in the error and work from there.

I do know that disabling the web server, disabling audio, and removing the 2 audio effects makes the error go away. The problem with debugging is that this device is mounted on the wall in my office a few miles from home. I have the same type of board at home, but it is brand new with nothing soldered to it. I have another board of a different brand that I use for testing my code at home. That just introduces more potential differences / points of failure. I will do more testing and report back.

I think the problem may be in the SerializeToJson() function in the PaletteFlameEffect class that starts at line 258 in include/effects/strip/fireeffect.h. Could you change the value 512 in line 260 of that file (AllocatedJsonDocument jsonDoc(512);) to 1024 and see what happens then?

Enabling WiFi, audio, and the other thing all take memory. You're seeing memory allocation failures. It's probably not a big with any of them as much as their general presence taking up space that the serializer wants.

Rutger has some changes in progress to make the serializer failure path a little better ...

On Thu, Oct 26, 2023, 3:36 PM Joe Schneider @.***> wrote:

@rbergen https://github.com/rbergen I clicked the wrong button when responding and accidentally closed the issue. So I reopened it.

NotifyJSONWriterThread is being triggered twice. On the remote I am using the AUTO button to either set the interval to 30 or 0. If it is setting to 30, I go ahead and trigger the next effect to give visual feedback that the button press worked. So, it writes the setting for current effect and then writes a second time for the change in interval.

Now that I know what methods are involved, I can dig a little deeper.

I changed 4 things and then discovered the problem: Enabled wifi and web server on the device. Enabled audio. Added two audio based effects to effects.cpp. Merged the latest changes from the main branch.

I will see which combo of those changes result in the error and work from there.

I do know that disabling the web server, disabling audio, and removing the 2 audio effects makes the error go away. The problem with debugging is that this device is mounted on the wall in my office a few miles from home. I have the same type of board at home, but it is brand new with nothing soldered to it. I have another board of a different brand that I use for testing my code at home. That just introduces more potential differences / points of failure. I will do more testing and report back.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/473#issuecomment-1781860289, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD33RTCEXGO5L56TCAMTYBLCWHAVCNFSM6AAAAAA6RXZWA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRHA3DAMRYHE . You are receiving this because you are subscribed to this thread.Message ID: @.*** com>

Actually, I now think that whole SerializeToJSON() function in the PaletteFlameEffect class is borked, and always has been. It becomes visible now because we switched to a newer version of ArduinoJson in #466, which is better in flagging buffer overflow conditions - which in turn relates to the comment @robertlipe just made. It seems I missed some effect serialization functions while implementing the changes he mentions, included in said PR.

@revision29 Could you replace lines 258 up to and including 267 in your /include/effects/strip/fireffect.h file with the following, and see what happens then with audio and the audio effects enabled?

    bool SerializeToJSON(JsonObject& jsonObject) override
    {
        AllocatedJsonDocument jsonDoc(1024);

        JsonObject root = jsonDoc.to<JsonObject>();
        FireEffect::SerializeToJSON(root);

        jsonDoc[PTY_PALETTE] = _palette;

        assert(!jsonDoc.overflowed());

        return jsonObject.set(jsonDoc.as<JsonObjectConst>());
    }

Note that this will not be the final, final version of this function. I will open a PR with an update to all SerializeToJSON() functions I missed before that will include the actual update to this specific function, but I'd first like to know if the problem you are seeing is indeed coming from it.

@rbergen I will try that in about 12 hours when I am back in my office.

Here's another piece of the puzzle. I got home and started to figure this out while using a different board. At home my primary testing board is a heltec wifi kit v3. When I set ENABLE_AUDIO to 1 it fails to build no matter what I try. Documentation says audio is limited to the m5sticks, but the code says it should work. The problem is the library / driver for processing audio is not being included at compile time and the code complains that the required functions are absent.

When I set ENABLE_AUDIO to 1 on my cheaper esp32 board, the same one defined in the basic demo config, it compiles just fine. But, I don't think it is including the library / driver for processing the audio either. I think VS Code is being inconsistent with its error checking and it should fail to build. What I think might be happening is that I am triggering effects that utilize audio, but the audio processing system is no where to be found. So the effects are returning errors, null, or undefined and is causing the json buffer problems when the effect's serialize to json method is being called.

The compile errors I am getting when compiling for the heltec board include: "'i2s_adc_enable' was not declared in this scope," "'i2s_adc_disable' was not declared in this scope," and "'I2S_MODE_ADC_BUILT_IN' was not declared in this scope"

I don't know if these issues are related or if this enable_audio failing to compile with some configurations is a distinct issue. Thought I would add some data points in case it's relevant.

That's enough clues for us to at least repro it, though it would have been easier if you'd just told us what you did and built.. At least now we have a better hint. If I turn on audio in DEMO for the heltecv3 builds, bad things happen.

I don't see compelling evidence of the code saying it should work. It was pointed out before that there's excessive copy-pasting in globals.h so there might be some dead code you see, but that board doesn't come with a mic and custom configurations in the code are pretty much up to the people that build that hardware combination.

diff --git a/include/globals.h b/include/globals.h index d8ecc07..84ffb18 100644 --- a/include/globals.h +++ b/include/globals.h @@ -244,7 +244,7 @@ extern RemoteDebug Debug; // Let everyone in the project know about it

define MATRIX_HEIGHT 1

 #define NUM_LEDS                (MATRIX_WIDTH*MATRIX_HEIGHT)
 #define NUM_CHANNELS            1

define ENABLE_AUDIO 0
define ENABLE_AUDIO 1

define POWER_LIMIT_MW 12 10 1000 // 10 amp supply at 5

volts assumed

pio run -v -e heltecv3demo

I do see errors including the one you cite. Investigating i2s_set_adc_mode as a starting place, we can see it's part of the doc in V4 of the native OS https://docs.espressif.com/projects/esp-idf/en/v4.4.6/esp32/api-reference/peripherals/i2s.html but if you look at version 5.0 https://docs.espressif.com/projects/esp-idf/en/release-v5.0/esp32/api-reference/peripherals/i2s.html those services are no longer available. If we dig into the current tree, we see it's provided in a path containing "deprecated" in the name, so that's not good. https://github.com/espressif/esp-idf/blob/b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e/components/driver/deprecated/i2s_legacy.c#L910

I'm not sure at a glance how that falls through the tangly #ifdefs in include/soundanalyzer.h, but you've just hit a combination that nobody's written the code for. I think that nobody has ever tried to turn on ENABLE_AUDIO for the https://heltec.org/project/wifi-kit-32-v3/ configuration - it doesn't have a mic anyway, so whomever needed that (which we've established is nobody in the time we've built on > ESP-IDF V5) would have had to build up a pin configuration, decide between an analog and I2S mic, and so on.

It's just not implemented.

We could link against the deprecated driver, but it looks like they have strong opinions about that: https://github.com/espressif/esp-idf/blob/b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e/components/driver/deprecated/i2s_legacy.c#L1914

Someone that cares about that configuration (it looks like it's not trivial. Sorry) "just" needs to reconcile the current ESP-IDF https://docs.espressif.com/projects/esp-idf/en/v5.1.1/esp32s3/api-reference/peripherals/i2s.html against include/soundanalyzer.h and friends. The API is just different. I can't tell if this code expects the Arduino interposer layer to take care of such differences, but it obviously doesn't.

It looks like you could probably draft code from https://docs.espressif.com/projects/esp-idf/en/v5.1.1/esp32s3/api-reference/peripherals/i2s.html#application-example into soundanalyzer.h and friends and have it work on any board. Well, you'd still need to invent a way for people to define their custom pinouts for .ws, .dout, .din, .mclk, and such.

I don't know if the trigger here is really "ESP32-S3" as much as it is "Any configuration that requires new ESP-IDF levels" as the newer chips are more likely to require/provide that.

The more basic boards just have simpler audio interface demands. The TTGO configuration builds, but it's a simple analogue MAX4466 mic. I2S mics are just more complicated.

I don't know which boards are your cheaper boards, but MY cheaper boards just don't have mics factory-installed; I attach them via analogue 4466's (cheap, low pincount) and not SPI. You'll just have to decide what combination of hardware you're really trying to build (that's "build" with a soldering iron AND "build" as in compile) and if it's something based on an exiting combination and turn that on or graft in code from the current ESP-IDF examples.

But this has nothing to do with the titular bug in this report.

I was actually kicking around an idea of using ESP-NOW to let a single NLD server chomp the audio, FFT it and just broadcast/multicast the NUM_BANDS elements of _Peak or other data so the other strands could "head" the data from the single strand with a controller with a microphone.

Of course, this would all have to be built...

But, yes, there's enough data here to confiirm that you've found a combination that just doesn't work with the code now checked in. That probably deserves a bugreport of its own.

Be prepared for it to set for a while since the number of people able/willing to work at that level of code on this project isn't high and the number of people wanting such a combination isn't a high number of people and the intersection of those two set is an even smaller number.

On Thu, Oct 26, 2023 at 10:51 PM Joe Schneider @.***> wrote:

@rbergen https://github.com/rbergen I will try that in about 12 hours when I am back in my office.

Here's another piece of the puzzle. I got home and started to figure this out while using a different board. At home my primary testing board is a heltec wifi kit v3. When I set ENABLE_AUDIO to 1 it fails to build no matter what I try. Documentation says audio is limited to the m5sticks, but the code says it should work. The problem is the library / driver for processing audio is not being included at compile time and the code complains that the required functions are absent.

When I set ENABLE_AUDIO to 1 on my cheaper esp32 board, the same one defined in the basic demo config, it compiles just fine. But, I don't think it is including the library / driver for processing the audio either. I think VS Code is being inconsistent with its error checking and it should fail to build. What I think might be happening is that I am triggering effects that utilize audio, but the audio processing system is no where to be found. So the effects are returning errors, null, or undefined and is causing the json buffer problems when the effect's serialize to json method is being called.

The compile errors I am getting when compiling for the heltec board include: "'i2s_adc_enable' was not declared in this scope," "'i2s_adc_disable' was not declared in this scope," and "'I2S_MODE_ADC_BUILT_IN' was not declared in this scope"

I don't know if these issues are related or if this enable_audio failing to compile with some configurations is a distinct issue. Thought I would add some data points in case it's relevant.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/473#issuecomment-1782246942, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD363PDUIZ4RJ3IR3YGLYBMVTZAVCNFSM6AAAAAA6RXZWA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBSGI2DMOJUGI . You are receiving this because you were mentioned.Message ID: @.***>

@revision29 I've actually opened #474 with the (proposed) final solution. If nothing else, it should unequivocally tell us which serialization process is driving things off the rails.

If I turn on audio in DEMO for the heltecv3 builds, bad things happen.

This is precisely why I stopped debugging anything that's specific to devices I don't own. There are too many variables to be able to do the debugging at a mental level only. In this case, when it concerns devices that do not have microphones built in, then the assumption should be that no effort has been made by past or current contributors to make them work with audio.

And that's where my involvement with this particular exploration/discussion ends.

@robertlipe I disabled audio and it breaks just the same as without it. That eliminates one possible point of failure. You mention excessive copy and paste. Yes. I have 2 devices in real world deployments. I have two different boards that I use to test various random things as I learn and experiment. That is on top of the already large number of devices defined in globals.

Regarding audio support. That is now a rabbit hole I will not go down. I thought it would be as "simple" as getting some non-serial mics from Amazon and wiring them to particular input pin. The mic would output a standard analog waveform. It looked like the audio code in NightDriver handles serial or analog input. But it looks like I am wrong. It will be much easier to source m5sticks, even though they are not available on amazon at the moment. My wild guess about audio support causing problems turns out to be, wrong.

@rbergen the code you pasted in a comment above did not fix the problem. I disabled the web server and the problem goes away. The project I have running the code does not need a webUI, so it's no big loss. At least now I can do a little more digging to see if the problem is memory or something else. What's crazy is I have a button that just triggers NextEffect and it does not trigger the problem. Only SetInterval does.

@revision29 The fact that NextEffect doesn't trigger the problem and SetInterval does makes sense. The latter triggers a JSON serialization of the effect list in EffectManager (and as such, all effects in it), the former only updates the "current active effect" index file.

On the other hand, disabling the webserver should not make a difference whatsoever. The IR remote code doesn't/shouldn't interact with it in any way, so why disabling that circumvents anything I can't fathom.

You could sync with the main branch again after #474 is merged, and then see what logging that generates when you retry.

@revision29 In the meantime, you could add some additional logging to see which effect fails to serialize. In effectmanager.h, within the function SerializeToJSON(), you could replace the following lines (line numbers 283 and 284 in the main branch):

            if (!(effect->SerializeToJSON(effectObject)))
                return false;

with these:

            if (!(effect->SerializeToJSON(effectObject)))
            {
                debugW("JSON serialization of effect \"%s\" with effect number %d failed!", effect->FriendlyName().c_str(), effect->EffectNumber());
                return false;
            }

and then rebuild, reflash and rerun. That should log name and effect number of the effect that causes the buffer increase loop - and indeed if it is the same effect every time.

I will be glad to merge and test.

It’s weird behavior for sure. If it were a memory issue the serializer should also break with NextEffect.

I just started throwing debug code all over the serializer. Set interval sets the variable and the notifies the json thread to write the config. As it builds the json document it successfully creates the interval object. The it iterates through the effects to build those objects. It fails to serialize the first object. That effect is EFFECT_STRIP_RAINBOW_FILL. I disabled that one. Now the first effect in my list is EFFECT_STRIP_PALLETTE. That one fails too So disabled all of the palette effects. The next one is EFFECT_STRIP_COLOR_FILL but does not break. I only had two effects, the other being strip fire. I could not get it to break. Two effects?Maybe I was running too many. So, I added 6 or 7 more color fill effects. Did not break. Then I set strip fire as the first effect. It did not break.

I then added back a strip palette effect. It broke again. Then I disabled that and added back the rainbow effect. It did not break. So, the problem has been narrowed to EFFECT_STRIP_PALLETTE. I suspect the problem is when the serializer is returning the result of jsonObject.set(…). It is successful on initial effect build. But when I am triggering SetInterval jsonObject.set is failing for EFFECT_STRIP_PALLETTE. Then the problem cascades and it starts to fail on the fire effect. Then it loops back through the effects and tries it all again, fails and the loop continues.

@rbergen I was off testing when you typed your debug code suggestion. I went a different route but got the same result. The effect that fails consistently is EFFECT_STRIP_PALLETTE (number 21). Sometimes I can get the fire effect (number 8) to fail.

I have been having trouble connecting to my device's web server. I don't know if that is a problem with the board running out of memory or if there is something with the network preventing me from accessing it. But, that is all the time for debugging I have for today.

@revision29

I have to stress not to jump to conclusions on this one. It's actually normal for the whole serialization process to have to go through a few cycles - where the overall serialization buffer size increases with 2048 bytes on each cycle - until the buffer becomes big enough to support the serialization of the entire list.

So, I recommend you re-enable all effects you started with and then see if there is at some point one effect that keeps failing to serialize, bringing the buffer increases to an infinite loop - which is what I understood the original problem to be.

If it is indeed EFFECT_STRIP_PALETTE that consistently fails to serialize, then that is the one I will focus on first.

I think everyone else has already walked out on this, so I'll at least include my notes on record in case someone else ever steps into this. This does not directly address the bug(s) that are observed here, but it's a study of the audio in path that explains what that code is doing and a bit of what that code almost certainly should NOT be doing.

NDL supports two styles of audio input at the hardware layer. The configuration of these is, well, challenging. They're hard to separate from the development board where it looks like they were first encountered, but as a guide to anyone attempting it, here are some notes. (I could be convinced to fluff this into an actual document in the source tree or a wiki or something as it comes from writing I did for another source)

For designers, the two choices in styles of mics are:

a common analog mic that delivers a DC (3.3 through 5VDC) coupled voltage that's biased at VCC/2 and swings up and down in voltage per cycle. We'll call this an MIC_ANALOG, just because it needs a name and the code doesn't really give it on. These are usually electret mics with adjustable gains via a trimpot. A small amplifier, like the Max4466 is commonly present. (Hint: that's a good term to use in a code search) These mics are super simple and usually have only three pins: VCC, GND, and OUT. You'll want to feed VCC the cleanest DC voltage you can get. If you're feeding it from something like a phone charger, a switching supply, or a buck/boost converter. Any noise you have on VCC represented as VPP is going to show on OUT as VPP/2 a the wideband switching noise is going to get dumped through. If you can't clean up the source, small caps on the mic may help. This isn't a course on reducing common-mode noise though; it's a discussion of why these mics are sub-awesome. Even if the switching frequency is above what you can hear (and what the spectrum analyzers in the code can graph) it's still a distraction that reduces the (literal) signal/noise ratio that the OUT line is swinging around and NOT representing the audio signal you care about. On the bright side, they're easy to hook up.
a mic using I2S. We'll cleverly call this one MIC_I2S. Electrically, I2S is like I2C, but optimized for sound. These are usually MEMS microphones, but the analog path in them is extremely short. There's a chip in the mic (sometimes on a board) that handles A/D conversion, signal conditioning and lots of other helpful things. It serves up 24-bit I2S packets and thus requires more pins and are usually 3.3V ONLY (not 5V tolerant) as the kinds of MCU's that have I2S busses are new enough they don't do 5V anyway. These are the class of mics used in home assistants like Alexa, Google Home, and similar products. You'll still need pins for VCC (3.3V, or as advised) and GND, of course. There's still an outut signal pin, but it's I2S and speaks in packets. Other pins will probably include SCK or CLK for the clock, toggling once per bit. Speeds can range between 512kHz through 12+Mhz - which is serious overkill for us. Word Select (WS) is an OUTPUT from the mic that drives a signal indicating whether this frame is for the left or right channel. How does the mic know if it's on the left or right side? There'a usually a pin labeled something like "L/R" that's an input to the mic. The designer ties it high or low to tell the mic whether to set WS to left or right during that part of the frame. So these mics need about six wires, (VCC + GND + L/R) for overhead plus (SD, SCK, WS) for carrying high speed data that has to be connected to the MCU. (Remember this requirement. It comes back...)

It looks like the three places configuration decisions are made (sigh) are src/main.cpp include/soundanalyzer.h and include/globals.h

include/globals.h:#define ENABLE_AUDIO 1 for either case Set INPUT_PIN to the pint number of an analog mic input. (Search that file for hazards and examples. Instead of defining them in the blocks for each build, there's a giant #if ENABLE_AUDIO that then tests the board name

src/main.cc:Configures the INPUT_PIN as an input pin, BUT does some special casing for a TTGO. It's a hack for someone's specific installation that drives one GPIO high (VCC), one low (GND), and uses the other as INPUT. (Searc for "TTGO") If you've used a build/configuration that sets TTGO, those pins are about to get clobbered.

This looks like a big runtime problem for later chips. GPIO 37 and GPIO38, in particular, are used by the SPI bus that holds external chips for RAM and flash. This sounds like a pretty icky problem to discover later. Having this in two different places makes it difficult to spot bugs. What if there's a TTGO that pasted LASERLINE definition? Pin37 is going to be reused a second time. That can't be good.

include/soundanalyzer.h: There is stealth configuration in this file. This is another place that all configurations are tested by name and some pins are hardcoded, ignoring entries in config files you may have meant to change. For example, if you're using a configuration that sets ELECTROW, you'll run into const i2s_pin_config_t pin_config = { .bck_io_num = 39, .ws_io_num = 38, .data_out_num = -1, // not used .data_in_num = INPUT_PIN ...and maybe those pins ARE where the bit clock and Word Select (see above) are set, but if you're in globals.h and scanning conflicts, this is a use of those pins you're not going to find looking in globals.h where most of the pin usages are defined. Why are they not using CONFIG_I2S_BCK_PIN and CONFIG_I2S_LRCK_PIN and such?

Worryingly, there are several blocks that are repeated. Some differ in what's clearly intentionally different ways, but some are different that just seem like they were copied from different generations of the same material. I knew I was looking for a problem with a mic configuration like we called ANALOG_MIC above. I knew they needed to set the i2s config bit I2S_MODE_ADC_BUILT_IN. There are two blocks that use this: TTGO/Mesmerizer/SpectrumWrover and "everything else". Unable to spot the difference, I threw them into tables and ran diff:

@.*** nightdriverstrip % diff -ub /tmp/a /tmp/b --- /tmp/a 2023-10-28 01:04:06.000000000 -0500 +++ /tmp/b 2023-10-28 00:48:50.000000000 -0500 @@ -4,7 +4,7 @@ i2s_config.dma_buf_len = MAX_SAMPLES; i2s_config.bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT; i2s_config.channel_format = I2S_CHANNEL_FMT_ONLY_LEFT;

i2s_config.use_apll = false;
i2s_config.use_apll = false, i2s_config.communication_format = I2S_COMM_FORMAT_STAND_I2S; i2s_config.intr_alloc_flags = ESP_INTR_FLAG_LEVEL1; i2s_config.dma_buf_count = 2;

It's POSSIBLE that's just an extremely subtle difference that justifies a repeated block. I'd wager that's a copy/paste/typing error. A comma in the rval instead of a semicolon?!?! In a 16 line block?

So it looks like the M5STACK/STICK/Core/Plus family uses MIC_I2S style of programming, with some assists in all three places. TTGO || MESMERIZER || SPECTRUM_WROVER_KIT and everything else (maybe?) are using the simpler MIC_ANALOG. We have some evidence, with our knowledge of perhaps more SoCs and boards, that a naive copy-paste when creating a configuration is possible to get pins changed that really shouldn't

Debugging all of this: Watch for a new line starting "Audio FPS" in the slow helper functions that's printed to the serial console a few times a minute. If everything is working, you'll see a low "MinVU" and a much higher PeakVU that changes from display to display. Until you can see those number bouncing around and reacting to noise in the room - and absence of that noise - there's no reason to dig into the effects until that works.

The problem at hand. It looks like some of the methods we're using in the I2S path have moved in the ESP32 S2 and S3 variations to reflect them being on different internal busses with the replacement API requiring less of this. For a vague hint, see https://github.com/espressif/arduino-esp32/issues/4707 and the solution seems to be to "update the code". For that access specifically, we can force it into the M5STICKC || M5STICKCPLUS || M5STACKCORE2 || ELECROW path and not use i2s_adc_enable, but I think that Joe's configuration has already left the rails. I think he's defining a MIC_ANALOG so messing around in the MIC_I2S path seems like a distraction. If you're on an S2 (unlikely since we require dusl-core) or, very likely C6, H2, or any of the other newer chips since then, it's worth steering it into the path in soundanalyzer.h that takes I2S_MODE_ADC_BUILT_IN - which gets us back to the weird duplicate function bodies.

Summary (Kind of - the above is fragments from several different pages of notes with different goals.)

Our configuration plumbing for setting up boards, memory size, strip configurations, pins used for buttons and extra peripherals, networking, effects management, and other features has been known to be come up short. Allowing custom mics is something I haven't yet seem others report, but this should be added to that stack off things to overhaul.

I'm with Rutger - without hardware in hand, an exact source tree that goes with a problem report, audio input in this project seems like a great system to never ever touch. Even then, I might only touch it with one of everything involved to be sure we didn't bring down others down. This is a special kind of mess. :-)

On Fri, Oct 27, 2023 at 11:09 AM Joe Schneider @.***> wrote:

@robertlipe https://github.com/robertlipe I disabled audio and it breaks just the same as without it. That eliminates one possible point of failure. You mention excessive copy and paste. Yes. I have 2 devices in real world deployments. I have two different boards that I use to test various random things as I learn and experiment. That is on top of the already large number of devices defined in globals.

Regarding audio support. That is now a rabbit hole I will not go down. I thought it would be as "simple" as getting some non-serial mics from Amazon and wiring them to particular input pin. The mic would output a standard analog waveform. It looked like the audio code in NightDriver handles serial or analog input. But it looks like I am wrong. It will be much easier to source m5sticks, even though they are not available on amazon at the moment. My wild guess about audio support causing problems turns out to be, wrong.

@rbergen https://github.com/rbergen the code you pasted in a comment above did not fix the problem. I disabled the web server and the problem goes away. The project I have running the code does not need a webUI, so it's no big loss. At least now I can do a little more digging to see if the problem is memory or something else. What's crazy is I have a button that just triggers NextEffect and it does not trigger the problem. Only SetInterval does.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/473#issuecomment-1783168037, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD37WUWIT4F4V4IFA7K3YBPMCVAVCNFSM6AAAAAA6RXZWA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTGE3DQMBTG4 . You are receiving this because you were mentioned.Message ID: @.***>

I have been working with using audio for led strip builds for some time now. Recently, I went back in time to a build before the code base changed. I put Atom light on a M5stick plus, and copying in strip effects from other projects, many of the effects were audio reactive, many more than what the current code base does. In further experimentation I tried some new things I have never tried before on the current code base. In trying to use M5 stack for led strip projects I noticed that if: -DSHOW_VU_METER=1 is added to build env it will output a vu_meter on 1 spoke. Not sure how it works, but this works on Heltec v1, M5 stack, and M5 stick. While many of the audio reactive effects draw blanks, this behavior is pretty consistent. That effect is sound reactive, and seems pretty accurate. That code is in drawing .ccp. Additionally, and this really blew my mind, I added #define NUM_INFO_PAGES 2 to the globals for Atom light. Low and behold it displays spectrum analyzer and bar graph to the oled display. That behavior appears to be in sync with the VU_meter effect. Unfortunately, it does not display current effect which is something I really wanted. While I was seeing VU values on serial out, wasn't sure if it was real or not. Apparently, it is, and audio is there, it is being processed, but for whatever reason it just isn't being pushed out onto the strips. Since I have a Heltec V1 on breadboard with mic, remote, I'll take that back into the old code base, and maybe TTGO, and see what works there.

@revision29 I'm reading back in this thread now, and I now notice this in the original bug report:

Power cycle the board and the effect interval now reflects the value triggered by the remote. The buffer increase has stopped.

I don't know how I missed this before, but that means JSON serialization of EffectManager with all effects must have completed successfully before the power cycle, otherwise the new interval won't be persisted - it's integrated in the EffectManager JSON blob that also includes the whole effect list. That just won't happen if the serialization process runs out of buffer memory.
That makes me wonder if the buffer increase loop is indeed infinite.

Can you provide a longer excerpt of your console logging that shows the JSON buffer size continues to climb (way) beyond the 10K your original bug report shows?

@rbergen I am deep into shoving debug statements all over the serializer codebase to find the issue. I suspect I am running into some hardware constraint of the board. If I add enough debug statements in some parts of the code it adds enough delay in the processing that it will not fail. It's weird. But, I have enough information to track down when and where it is failing. My goal is to see if I can provide a fix or optimization.

Some brief observations. When the serialization succeeds, the size of the whole json object is using about 2134 bytes. The failure happens when the json for a specific effect is set into the effects json object. Each effect has a limit of 192 bytes. The buffer size for the json object is not being overrun. When it fails, the loop of buffer increase is infinite. Out of humor, I was going to paste in about 10 seconds of buffer increase statements, but the VS Code monitor buffer could not hold all of the data. At some point drawing the LED pixels gets very, very laggy.

I'll keep poking around and eventually propose a fix. If I need more help, I will reach out. I now have a very good understanding of the whole json serializer system.

This is about 5 seconds of buffer increases: (W) (NotifyJSONWriterThread)(C1) >> Notifying JSON Writer Thread (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 6144 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 8192 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 10240 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 12288 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 14336 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 16384 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 18432 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 20480 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 22528 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 24576 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 26624 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 28672 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 30720 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 32768 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 34816 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 36864 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 38912 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 40960 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 43008 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 45056 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 47104 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 49152 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 51200 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 53248 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 55296 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 57344 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 59392 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 61440 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 63488 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 65536 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 67584 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 69632 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 71680 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 73728 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 75776 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 77824 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 79872 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 81920 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 83968 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 86016 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 88064 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 90112 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 92160 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 94208 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 96256 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 98304 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 100352 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 102400 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 104448 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 106496 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 108544 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 110592 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 112640 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 114688 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 116736 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 118784 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 120832 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 122880 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 124928 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 126976 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 129024 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 131072 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 133120 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 135168 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 137216 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 139264 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 141312 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 143360 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 145408 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 147456 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 149504 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 151552 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 153600 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 155648 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 157696 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 159744 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 161792 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 163840 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 165888 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 167936 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 169984 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 172032 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 174080 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 176128 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 178176 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 180224 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 182272 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 184320 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 186368 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 188416 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 190464 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 192512 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 194560 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 196608 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 198656 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 200704 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 202752 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 204800 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 206848 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 208896 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 210944 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 212992 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 215040 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 217088 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 219136 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 221184 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 223232 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 225280 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 227328 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 229376 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 231424 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 233472 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 235520 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 237568 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 239616 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 241664 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 243712 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 245760 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 247808 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 249856 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 251904 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 253952 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 256000 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 258048 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 260096 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 262144 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 264192 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 266240 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 268288 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 270336 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 272384 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 274432 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 276480 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 278528 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 280576 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 282624 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 284672 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 286720 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 288768 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 290816 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 292864 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 294912 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 296960 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 299008 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 301056 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 303104 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 305152 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 307200 bytes (W) (SerializeWithBufferSize)(C0) Out of memory serializing object - increasing buffer to 309248 bytes

@revision29 Yep, that looks like an infinite loop to me.

It's a weird error state indeed. ArduinoJson's JsonObject::set() should really only return false if the containing JsonDocument's buffer is too small - or that's how it worked till very recently at least.

@rbergen after a busy couple weeks mixed with a weeklong trip for a wedding, I dove back into the failure. The failure is happening in the SerializeWithBufferSize function within jsonserializer.cpp. The failure is not the code. What is happening is when it calls pJsonDoc.reset(new AllocatedJsonDocument(bufferSize)); it fails to allocate physical memory. I am guessing the device is running out of memory and it will never be able to complete the operation. That was my assumption before diving deep into the code. My goal was to get the code to fail gracefully instead of entering an infinite loop. I have achieved that and will submit a pull request in the next few days. The real solution is to buy boards with more memory, but a software fix is also advisable.

Of note: it is failing before the json serializer for the effects manager is called. It passes the failure into subsequent function calls. The debug code made it look like an effect's serializer was failing, but that was not the case. It could be that an effect is using an obscene amount of memory causing the system to run out. But, I don't yet have the skillset to be able to trace that out.

I have submitted a fix for the infinite loop: pull request 499. It was a fun adventure figuring out the complexity of the serialization process. The root cause was not in the effects manager or the serializers for each individual effect. It was a memory allocation problem. The code did not handle that failure point. If there was not enough memory for the JSON document, it would fail to create the variable in memory. Any attempts to add data to it, such as effects definitions, it could not. The code assumed the failure was too small of a buffer so it would increase it. On the next loop it would fail again. If there was not enough memory the first time, increasing the buffer by 1024 (or whatever it is) is NOT THE SOLUTION. "You don't have 4kb available? Okay, give me 6kb." Doomed to fail every time. Despite the dozens of hours spent looking for the problem, I learned a lot.

I have relevant skills, but little patience. If you think some effect is being naughty and can produce a reproducible test case, I'll help track it and whack it.

But, yeah, with 8MB RAM/16MB Flash boards going for $5.20, I totally understand the urge to flee instead of fight, even knowing that if there's a leak it'll catch up with you eventually.

Fragmentation may be our enemy here, but let's get a failing test case under a microscope before speculating too much.

On Fri, Nov 17, 2023, 4:24 PM Joe Schneider @.***> wrote:

@rbergen https://github.com/rbergen after a busy couple weeks mixed with a weeklong trip for a wedding, I dove back into the failure. The failure is happening in the SerializeWithBufferSize function within jsonserializer.cpp. The failure is not the code. What is happening is when it calls pJsonDoc.reset(new AllocatedJsonDocument(bufferSize)); it fails to allocate physical memory. I am guessing the device is running out of memory and it will never be able to complete the operation. That was my assumption before diving deep into the code. My goal was to get the code to fail gracefully instead of entering an infinite loop. I have achieved that and will submit a pull request in the next few days. The real solution is to buy boards with more memory, but a software fix is also advisable.

Of note: it is failing before the json serializer for the effects manager is called. It passes the failure into subsequent function calls. The debug code made it look like an effect's serializer was failing, but that was not the case. It could be that an effect is using an obscene amount of memory causing the system to run out. But, I don't yet have the skillset to be able to trace that out.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/473#issuecomment-1806623724, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCSD36MALINEXPFFJU6Q4DYE7P3HAVCNFSM6AAAAAA6RXZWA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGYZDGNZSGQ . You are receiving this because you were mentioned.Message ID: @.***>

@robertlipe I don't remember what I have said in which context, so I will summarize here. It turns out that no effect was causing any issue. The root cause was a failure to allocate memory for the jsondoc that holds all of the serialization data that will be written to the effects file. That failure happens before any effect is serialized. It then passes the failure to each effect as it serializes. The last PR I submitted mitigated the failure. So, the infinite loop has been patched. But, it still does not deal with the root cause of a lack of memory. This is a problem I want to address later. I have another set of changes I am working on that is taking my focus. After that, I will need to get debugging working correctly in VS Code to see if I can determine what is using memory. That part is a little over my head for now.

PlummersSoftwareLLC / NightDriverStrip

Programmatically changing the effect interval causes an infinite recursive json buffer increase #473

Bug report

define MATRIX_HEIGHT 1

define ENABLE_AUDIO 0

define ENABLE_AUDIO 1

define POWER_LIMIT_MW 12 10 1000 // 10 amp supply at 5