ErgoDox EZ Shine - Left half RGB update lag and possible solution

[Taudris]

I've found that there is a significant amount of lag when sending LED data to the left half, enough that I can't use it for anything beyond its current purely-decorative functionality. (I was hoping to use it to indicate the current layer, but it's too jarring to press a key and see a cascade from right to left.)

I found that sending the LED data a second time causes an immediate update, so I modified the bit bang code to send the LED array a second time using a repeated start, except for the last LED. The last LED continued to lag. Thus, it is necessary to send the full LED data twice.

The schematics show that there's an FPGA in the left half which controls the LEDs. Maybe its firmware has some kind of per-pixel double buffering issue where it reads from the wrong buffer? The source code for it is not available to my knowledge, so I can't examine it (I don't know the first thing about FPGA programming anyway, though I'm somewhat curious about it), and it's not user-updateable without voiding the warranty, so unfortunately, I think sending the LED data twice is the only solution to the lag until ErgoDox EZ fixes the FPGA firmware and starts putting it in production units.

When not sending the LED data twice, there are occasions where the LEDs update instantly, but they are rare. Maybe other activity on the I2C bus is triggering a sooner update somehow? Could just be a race condition in the FPGA firmware. I might try some tests with I2C matrix scanning disabled to see if this changes.

I'm also curious as to why the left half LEDs are bitbanged rather than sent using the hardware peripheral. Is there a communication speed limitation in the FPGA?

[drashna]

I personally use the underglow as a layer indication, with animations too. I don't see this lag that you're referring to.

I mean, I do see a slight delay, but in the MS range (less than 20ms, if I had to guess). So. it's not noticeable, unless you're sitting and waiting for it.

Do you have another TRRS cable to use, maybe?

[Taudris]

I already tried slowing down the I2C communication by increasing I2C_DELAY from 1 to 10, with no difference. I doubt it's the cable. I don't have another one I could test with anyway.

The delay I'm experiencing is closer to 100ms. It's enough to throw me off/feel wrong, and it's not something I have to look for to see.

For one of my layers, when you tap to toggle it (rather than press another key), I coded up an animation that includes occasional strobing with 100ms pulses. The two sides appear to alternate during the strobing. I'm also able to emulate this manually with my momentary layer keys.

I also saw this behavior with the stock FW, just turning the lighting off and on. It wasn't an issue for me then, since it was purely decorative.

I'll try to post a couple short clips later, with and without the fix.

[Taudris]

Another thing occurred to me. I did one test last night with a 1 second delay after writing the left half LEDs, and the LEDs did not update until after the delay passed. This suggests that the FPGA has a 1 frame lag with no useful time limit. As to why this 1 frame delay is normally shortened to a few tens of ms, I think these are the possible causes:

1. rgblight_set() could be getting called in a way I'm not yet aware of. I'm probably going to check this by adding a delay to it and using the 3 single color right half LEDs as a counter.
2. Other I2C activity (matrix scanning) could be causing the FPGA to present the current frame sooner than the next frame. I'm going to try disabling I2C matrix scanning and see what effect that has on the frame lag.

[Taudris]

So it seems like this is an issue of configuring the I2C peripheral or the pins. I reduced init_mcp23018() in ergodox_ez.c to this:

    mcp23018_status = 0x20;
    if (i2c_initialized == 0) {
        i2c_init();  // on pins D(1,0)
        i2c_initialized = true;
        _delay_ms(1000);
    }
    //i2c_start(I2C_ADDR_WRITE);
    //i2c_stop();

And the lag is gone, even sending the LED data only once. Uncomment the start and stop, regardless of the address passed to i2c_start(), and the lag comes right back.

I haven't been able to find a real fix for this. I don't know why sending the LED data a second time seems to help.

The TWI bitbang code clears TWEN before using the pins, so I don't know what else needs to happen for the pins to become usable. Adding 10ms delays before and/or after the bitbanged start/stop makes no difference. I'm wondering if the TWI pins actually return to their power on state when TWEN is cleared... Maybe there is another register that needs to be touched. (This is my first experience with AVR.)

[Taudris]

Video of the issue. Matrix scanning disabled is lag-free, matrix scanning enabled is laggy (it's somewhat exaggerated in the video because I added a delay for testing, but you get the idea). Also note the 3 single color LEDs counting with each press; this counting is done in a CUSTOM_DRIVER rgblight_set(), and you can see that the only increments are from me pressing and releasing the layer key.

[Taudris]

I get the same behavior when setting LEDs using the TWI peripheral instead of bitbanging.

I tried the above technique for disabling slave scanning, then putting i2c_start(I2C_ADDR_WRITE);/i2c_stop(); in a few spots in my rgblight_set(). The only theory I have at this point is the FPGA firmware has some kind of issue when I2C traffic for other devices is present.

Anyway, rather than write all the LEDs twice, it seems to be sufficient to send a start, the first byte of LED data, a stop (these steps do not cause any LEDs to change color when I2C matrix scanning is enabled, but it DOES trigger a change when I2C scanning is disabled), and then send the LED data.

[drashna]

Wow, that is pretty bad. For reference: https://photos.app.goo.gl/H0QGtXVT1hD58LrL2

That said, have you tried playing with QMK_KEYS_PER_SCAN. That may help with the issue you're seeing? (or not) https://github.com/qmk/qmk_firmware/blob/4764e7712109be63df6395d70b069bceb0258e61/docs/config_options.md#behaviors-that-can-be-configured

Also, have you tried pinging Erez from Ergodox EZ? If not, it may be a good idea, as he's pretty awesome and super helpful.

[Taudris]

Yeah, I can't see any lag at all on yours. I wonder what the difference is.

I doubt QMK_KEYS_PER_SCAN would help in this case, as I'm pressing just one key to switch layers. Though it might be good to turn on anyway.

I wonder if the issue would still present itself if I tried disabling slave scanning and sending a non-existent address value. Or sending a start/stop pulse with no address.

@ezuk, get in here! (Does this summon him?)

[ezuk]

It summons me -- but it doesn't make me know C. ;) If the board was running on Ruby or JavaScript I'd be a little more useful, but for all things C I defer to the one and only @jackhumbert .

[drashna]

@Taudris could you post the full function, or a link to your modified ergodox_ez.c file?

And other than fixing the lights, does it cause any issues?

[Taudris]

Here's a file that I added to my keymap. I'm calling lighting_task() from matrix_scan_user(). This code depends on #2350, though it should be easy to remove that dependency (probably just need to change the call to ws2812_setleds() and remove the gamma correction). This code also requires you to put RGBLIGHT_CUSTOM_DRIVER = yes in your makefile since it defines a new rgblight_set(). The actual lag issue is worked around in i2c_setleds().

I haven't seen any other issues arise from this. I've been using it at least every workday since I came up with the workaround on 1/22.

[drashna]

I don't think that this is still an issue? There has been a lot of changes and improvements to the rgb code since this, so I think this may be fixed

[Taudris]

It does seem to be fixed now. I'll reopen if I see this issue again in the future.