fredizzimo / infinity_ergodox

Unofficial keyboard firmware for Input Club's Infinity Ergodox
GNU General Public License v2.0
61 stars 16 forks source link

There's no LED support #5

Open fredizzimo opened 8 years ago

fredizzimo commented 8 years ago

I didn't solder LEDs to the keyboard myself so I'm unable to test and implement this feature myself. It shouldn't be hard to do, using the visualization framework.

If you either want to test or implement this, please contact me.

kejadlen commented 8 years ago

I'd be happy to test this out. Would be interested in implementing it too, but not sure at all where to start.

tthayer commented 8 years ago

I'd be willing to test some things out too. Just need some guidance.

fredizzimo commented 8 years ago

Hi,

Sorry for not replying earlier. I have been and still am on a trip, so I haven't had time to answer and think about this.

But today I had some time in the hotel room to look more into it, It doesn't seem to be as easy as I first thought, but it shouldn't be too hard either. I think the design that makes most sense would be to expose the LEDs as gray scale display that can be controlled the same way as LCD. Each pixel would represent a led, and the color would represent the brightness.

The advantage of doing it this way, rather than something custom would be that then you could use the drawing routines of uGFX, including font rendering. Another advantage is that the same abstraction would work for keyboards with RGB leds as well.

In order to do this, a new uGFX display driver would need to be made, just like the LCD driver (in the drivers folder of the main repository). This driver would have to use the I2C functionality of the Chibios to communicate with the LCD controller according to the specs. http://www.issi.com/WW/pdf/31FL3731C.pdf.

Some tweaking of the makefiles and the chconf is needed, to enable the build of the display, and to enable multi-display support (which works slightly differently than for using just a single display) for uGFX.

A boardfile would also need to be made (also in the drivers folder), that enables the right input and output modes for the pins, to support the I2C communication

Then this would need to be integrated into the visualizer library like the LCD.

I think I can get something for you to test quite soon after I'm back. But I would of course be very happy if the instructions here are enough to get you started in actually implementing itself.

fredizzimo commented 8 years ago

Hi @kejadlen, @tthayer.

I have now implemented the first version of LED support. But I have no idea if it actually works. The only thing I know is that it doesn't crash, and the rest seems to work OK.

So it would be great if you could test it out.

I have submitted my changes in the led branch, so to take it into use you have to do this

git fetch origin
git checkout -b led origin/led
git submodule sync --recursive
git submodule update --init --recursive

Then make and flash both halves. When the keyboard starts you should see a looping test. It would be great if you could take a video or something, so I can see that it works properly, but it's ok if you just verify it yourselves too.

The test should go like this

  1. Fade in all LEDs to full brighness in 1 second
  2. Show all leds for 1 seond
  3. Fade out all LEDs so that they are completely shut down in 1 second
  4. Start to fade in a gradient pattern, the outside should be the brightest, and the middle the lightest, in one second
  5. Move the pattern from outside to inside for 3 seconds
  6. Crossfade the pattern to another one that goes from top to bottom in one second
  7. Move the pattern from top to bottom in 1 second
  8. Crossfade to a pattern that goes from inside to outside in 1 second
  9. Move the pattern from inside to outside for 3 seconds
  10. Crossfade to a pattern that goes from to to bottom in 1 second
  11. Move the pattern from to to bottom for 3 seconds
  12. Fade out all leds in 1 second
  13. Start over

You should report anything that isn't as you think it should be. Especially if some or all LEDs are not working, or if the patterns look wrong.

The pattern implementation is in tmk_visualizer/led_test.c And the actual low level communication mostly in drivers/gdisp/IS31FL3731C

tthayer commented 8 years ago

Nothing is happening with the LEDs. The display backlight on the right-hand half cuts out after a minute and doesn't come back on until the keyboard is replugged in.

fredizzimo commented 8 years ago

Thanks for the report. Guess I have to re-check all the code.

But the fact that the backlight cuts off is very strange. It doesn't happen on mine with no leds installed.

Does the keyboard still work if the backlight is off? And does the LCD go blank too?

tthayer commented 8 years ago

Thanks for working on this! The keyboard continues to function just fine and the LCD continues to cycle through the layers display. 

Sent from Outlook for iPhone

_____________________________

From: fredizzimo notifications@github.com Sent: Sunday, April 24, 2016 12:32 Subject: Re: [fredizzimo/infinity_ergodox] There's no LED support (#5) To: fredizzimo/infinity_ergodox infinity_ergodox@noreply.github.com Cc: Tony Thayer tony.thayer@gmail.com, Mention mention@noreply.github.com

Thanks for the report. Guess I have to re-check all the code.

But the fact that the backlight cuts off is very strange. It doesn't happen on mine with no leds installed.

Does the keyboard still work if the backlight is off? And does the LCD go blank too?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

dervos commented 8 years ago

Perhaps as a 'hello world' test you could hook one led port up to a volt meter. It would allow for easier initial testing and you'd be able to verify the test results of others.

fredizzimo commented 8 years ago

@tthayer, I found an issue, there was an off by one error in the code, which had been left by a refactoring I did.

The code was taking care of setting which LEDs are enabled to the hardware. The controller has two blocks, and only one of them is mapped to the keyboard LEDs. And of course the byte layout for those are interleaved, which means that no LEDs would get mapped. So that would explain why nothing shows up.

But this does not explain why the LCD backlight stops working. Although both are LEDs they have nothing to do with each other, and controlled by entirely different controllers. A memory corruption of some sort could be possible, but I doubt it, especially since most values should be re-written when you switch layers. I also think that this would happen on my board without LEDs as they should not affect the software at all.

Some kind of USB power issue could also be possible, if it detects that there's not enough power and shuts the LEDs off. But that also sounds very unlikely since the LEDs were not working and they shouldn't draw any power.

So for now, the most likely cause sounds like there might have been some kind of inference because of the incorrect mapping. So I suggest that you try my fix out.

@dervos Thanks for the suggestion, I guess a multimeter would show at least something, even if the LEDs are PWM controlled. I will try it, if this fix isn't enough, and there are other bugs. I don't want to open the keyboard if it's not necessary, as the acryllic is very hard to align right, at least on my keyboard. I have the first version, where they didn't fix the space between the thumb section, so I have less than one mm of space there.

tthayer commented 8 years ago

@fredizzimo The LEDs still don't power on but the right-hand LCD backlight stays lit now.

fredizzimo commented 8 years ago

@tthayer, thanks for the report, although it's completely the opposite from what I expected.

I will continue reviewing my own code, and test with a mult-meter before I ask you again.

tthayer commented 8 years ago

@fredizzimo No problem. I wish I was able to provide better feedback!

fredizzimo commented 8 years ago

@tthayer, I just pushed a new version, that forces all the LEds on directly from the driver, bypassing all visualization logic.

I get some readings when testing with the multimeter, but it varies from LED to led for some reason. Maybe you could test this to see if some of them light up?

kejadlen commented 8 years ago

@fredizzimo Unfortunately, I'm away from my Infinity until Thursday, but looking forward to helping out with testing and debugging once I get back.

tthayer commented 8 years ago

@fredizzimo That worked! They're all lit now. The right side now feels unresponsive and kind of slow though. Keys on the right side are also 'sticking'.

fredizzimo commented 8 years ago

@tthayer, Great. Do they all have the same brightness?

The unresponsiveness probably comes from the fact that the animation is still running in the background, and the same problem actually happens for the LCD screen too. I will have to profile and optimize some things.

@kejadlen, Don't worry, I have the help I need for now.

I will check if I can find the reason why it doesn't work through the visualizer. Probably not for too long though, as I have to go to bed quite soon.

tthayer commented 8 years ago

Yes, it appears they all have the same brightness.

fredizzimo commented 8 years ago

Hi @tthayer, @kejadlen.

I have fixed a couple of issues, and I think that the LED animation should be working now. Could you test it, please?

The brightness probably don't change as linearly as it should since I didn't apply any gamma calculations, as described in http://www.issi.com/WW/pdf/31FL3731C.pdf, read the "gamma correction" section. I'm not sure if I should apply that correction on the user side in the visualization code or, as a helper function that can be called, on the driver side, done automatically. Doing it on the user side would be more flexible, but I'm not sure if that kind of flexibility is needed. What do you think?

kwkroll32 commented 8 years ago

@fredizzimo Do you still need help testing this? I just flashed the LED branch and it seems to be running the light test loop.

fredizzimo commented 8 years ago

@kwkroll32, thanks. Yes, @tthayer and @kejadlen haven't had time to test the latest version yet.

It would be nice if you could verify that everything is done in the correct order, like I described above. And that all LEDs light up exactly as described.

I'm also interested in the linearity of the brightness, since there's no gamma correction as described above. How bad is it?

Once everything is confirmed working, I guess I will disable the demo loop, and maybe replace it with a fixed brightness settings, or even all LEDs off. You should then be able to customize it the same way as the LCD screen. I will also merge it to the master at that point.

tthayer commented 8 years ago

I'll try it out this weekend.

kwkroll32 commented 8 years ago

I uploaded a short video of the LED behavior. Hopefully you can tolerate the poor quality. https://youtu.be/-M3CBAGVVEY

fredizzimo commented 8 years ago

@kwkroll32, thanks for the video.

It looks like gamma correction is definitely needed. And there also seems to be some bugs, for example at around 0:08, it looks like half of the keyboard stays fully lit, while the other half is off.

I will look at it maybe later today or tomorrow, but if some of you want to have a look too, the bug should be somewhere in the tmk_visualizer/led_test.c file.

kejadlen commented 8 years ago

@fredizzimo Having trouble updating ugfx:

fatal: reference is not a tree: e221a690616e20f87e0b0088baffdbd5427be862
Unable to checkout 'e221a690616e20f87e0b0088baffdbd5427be862' in submodule path 'tmk_visualizer/ugfx'
Failed to recurse into submodule path 'tmk_visualizer'

Edit: Nevermind, I figured out the problem - git was still using Tectu instead of fredizzimo for the remote.

bremoran commented 8 years ago

Once my LEDs arrive, I'm happy to help out with testing this feature out.

fredizzimo commented 8 years ago

Hi All,

I have hopefully fixed the gradient code. I don't know what I was thinking when I wrote the original function, it seemed like I had only done it halfway.

The effect should be much better now, with this fix applied. But to make sure it would be nice if someone could test it.

I still think the gamma correction is needed though, so I will implement that next. It will be done inside the driver so it should be completely transparent to the user.

kwkroll32 commented 8 years ago

That looks a lot better; new video below. I had a hard time discerning what it was supposed to do last time. The only thing I'm not sure about is whether the horizontal crossfade is supposed to start in the middle column. You can tell that it starts one away from the middle column in both the left-->right and the right-->left fade. https://youtu.be/lGJzUa6dBUE

fredizzimo commented 8 years ago

Thanks, that really confirms that brighness correction is needed.

I also think that there might be some problems with the top row, it doesn't look like it's having the same brightness as the other rows from the video.

Also the thumb cluster, seems to be acting strangely, the outer keys seems to be quite bright. And it gradient is supposed to go down below the main keyboard rows to the two thumb buttons, because I represent the keyboard as a 7x7 bitmap, where the two upper thumb keys are part of the bottom main keyboard row.

I thought that this gradient effect would be enough to diagnose that all LED works individually correctly, and it might be, when the brightness correction works. But I still think I have to add one step that lights up each led, one by one, for easier diagnosing.

kwkroll32 commented 8 years ago

I've been reading the code and documentation to figure out how this works [having written lots of Python, but never working with C or hardware controllers]. One thing I noticed is that commenting out portions of the animation test in led_test.c can break the entire animation altogether. Do you know what that's about? I also commented out the corresponding lines from led_test.h, and reduced the number of frames to match the number of frame_functions.

In other words, how might I limit your animation test to [keyframe_fade_in_all_leds, keyframe_no_operation, keyframe_fade_out_all_leds]? I think that having these 3 as a sanity check, plus 1 complicated test like the crossfade, might make it easier for me to troubleshoot.

fredizzimo commented 8 years ago

Hi @kwkroll32,

I'm not very happy with the ease of use of the code myself either. I'm a C++ programmer, with very little experience in C, so I'm not very good at making good C interfaces. There's also special requirements for hardware controllers like this, like avoiding allocations, which complicates things further.

Furthermore I want to extend it with more features in the future, like being able to play animations on top of each other, a bit like layers. That would be useful for things like adjusting the brightness, while you are adjusting the brightness value would be showing, and when it's done, the display (LCD, LEDs and backlight) would resume to what it was showing before.

Therefore I have been thinking of doing a C++ interface, which would be easier to use, with less chances of making mistakes that the compiler can't detect. Originally I used C for the Visualizer because the TMK firmware is otherwise written in C, and I was also hoping that this would become a more official part of it. However that's not the case, so I could use C++ instead.

The documentation might also be lacking a bit at the moment, I will try to improve that once we have figured out what your problem is.

That said, you should easily be able to remove keyframes from the example code. You should only need to comment out or remove the frames you don't need, and change the keyframe count. But remember to always also update the list of frame lengths to match. You don't need to remove the code that actually defines those frames.

So the simple case you are talking about would look like this

keyframe_animation_t led_test_animation = {
    .num_frames = 3,
    .loop = true,
    .frame_lengths = {
        MS2ST(1000), // fade in
        MS2ST(1000), // no op (leds on)
        MS2ST(1000), // fade out
    },
    .frame_functions = {
        keyframe_fade_in_all_leds,
        keyframe_no_operation,
        keyframe_fade_out_all_leds,
    },
};

Even better, you could define your own animation in visualizer_user.c, just copy the animation example struct, but give it another name. And then start that animation instead from the function initialize_user_visualizer. Or you could of course bind different animations to different layers, and start them from the update_user_visualizer_state

If you do it that way, you can leave the led_test functions completely unchanged.

You can of course also write your own keyframes, since these ones are more of a demo nature, than useful on a real keyboard. For that you probably have to check out the ugfx documentation

It also probably helps to increase the keyframe lengths, if you want to see what it's doing a bit better. I had to record the video with VLC, and then step frame by frame, to analyse it.

If this doesn't help you, you can post some code, and I will hopefully be able to tell you what the problem is.

kwkroll32 commented 8 years ago

Thanks for all the helpful info. Strangely, I still can't get my simple, 3-frame example to run. First, I tried deleting all the other frames and changing the frame count to reproduce the code you pasted here. When that didn't work, I reset all my files, inserted a new animation struct containing this code, and modified visualizer_user to use it instead of your test. In both cases, the keyboard boots up fine but won't run turn on any LEDs. I'll look more closely this weekend, cutting 1 frame at a time until the led test breaks.

Regarding C vs C++, I guess I'd encourage you to do what's comfortable if there's no chance for TMK to adopt this officially. How much of the existing code would need to be rewritten in C++; just the visualization stuff, or a lot more?

fredizzimo commented 8 years ago

I can probably help you more, if you share the code that you have. Either by pasting it here, or pointing me to Github branch with your changes.

The change to C++ would be more of a simple refactoring, than a rewrite, and would only be limited to the Visualizer, which is very small. So from my part it's just a few hours of work, no big deal.

It would break the user_visualizers though, so that's one thing to consider. I don't know how much people have been customizing it, but probably not that much. Also the transition from the old code to new, should be a quite straightforward transformation. So I don't think that's a too big issue either.

I also have plans for making an emulator, which can be run on the PC, where you can test all your animations, and how they interact with each other. This would save a lot of time when writing custom visualizations, and also save the life time of the controller. The specs say that the minimum life time of it is 10000 flash uploads, with the typical of 50000. To me 10000 sounds quite low, especially if you are tweaking a lot of Visual Effects.

fredizzimo commented 8 years ago

Hi All,

I have now added support for linear brightness calculation. So I think the effects should look a bit more sensible now.

I haven't added the new animation that I mentioned earlier, the one that would light up each led one by one. That will probably happen tomorrow, if it's needed for verifying that the leds work correctly, since I want to debug the screen corruption bug a bit more before going to bed.

F1LT3R commented 8 years ago

I'm also very interested in trying this. Where can I start?

ec1oud commented 8 years ago

Your LED branch works for me! Thanks for figuring it out! So basically your firmware is now much more useful to me than the official one, which is constantly resetting and making typing next to impossible.

So the next thing I'm interested in is just to make the backlighting practical: have a default brightness, and have keystrokes for adjusting the brightness. So far I haven't thought of much use for visualization, but I suppose there are two cases: it should be controlled via software running on the PC, or it should interact with the keystrokes somehow - show a ripple effect away from each key as it's pressed, or something. But simply having controllable brightness will be a good first step.

I have Massdrop's Vortex Backlit Doubleshot PBT Keycaps on most of the keys. But it's not a complete ergodox set, so I have plain clear keycaps on a few keys. (It would be interesting to be able to get the remaining ones with proper legends some day, but my layout is not a normal Ergodox, so I'd need custom ones.) The clear caps are too distracting if they are too bright, whereas the Vortex ones look better with high brightness, because much less light gets through. So being able to set a pixmap for the default brightness is going to be useful. I got that done with a custom build of the official firmware, finding each key by trial and error. But that firmware is not reliable enough to use.

fredizzimo commented 8 years ago

Hi @ec1oud,

Yes, those things are next on my list as well. As soon as this LED branch is confirmed working well enough, I will merge the the support to the master branch.

Of course the current demo effect, needs to be replaced by something more practical first, like you said. But first I need to confirm that it really looks as it should. I have a couple of fixes for the demo effect, that I soon will ask people to test.

For the brightness control, which the LCD screen also needs. I have been waiting for the official support for the new action mapping system by TMK, which would make mapping custom functionality to keys much easier. There's also a minor problem of how to store the brightness in the EEPROM, since the slots are allocated by the tmk_core, and @tmk doesn't really wan't any keyboard specific functionality there...

At the first stage, I think the LEDs should interact with the keyboard itself only, as there's no host to keyboard communcation in the TMK firmware currently(at least not that I know of). It's already possible to react to layer changes from the user visualizer, but I will add support for single keypresses, or combinations as well, and of course mapped keys.

There's also another cool thing related to the visualization coming up, and that I have almost completed. I have made a PC based rendering, which simulates the keyboard LCD and LEDs, and render them on the screen. This makes it much easier to create visual effects, since you can run it in a debugger, and don't have to upload it to the keyboard. The long term goal for this is to have it communicate with the keyboard, so that you could basically simulate the whole keyboard, and maybe even change mapping, without having to upload a new firmware. You just need to recompile the PC version. The same software could also be used for communicating things from the PC to the keyboard.

But for now, it's only visualizing stuff, and currently there's only Windows (cygwin) support. You can find it in the https://github.com/fredizzimo/infinity_ergodox/tree/visualizing_emulator, branch.

fredizzimo commented 8 years ago

Hi All,

I have now updated the led branch with the fixes.

It would be nice if someone could capture a video to confirm that everything is working.

I hope the update rate is good enough, since the new effect requires 7 cosine calls per update. There's no hardware floating point on the chip, so it has to be done in software and can be quite slow. Also the update of the LCD, Link and LED hardware is quite slow, which reduces the available cycles further, since there's currently no DMA support in the ChibiOS system, which the keyboard uses. If the cosine turns out to be too slow, it would be quite easy to use a lookup table or approximation.

The branch now also contains the emulator that I mentioned above. I haven't written any instructions on how to use it yet. But it should be quite straightforward to compile it in Cygwin using the Makefile.emulator. You need to build http://www.glfw.org/ first, in a parent folder of the keyboard repository(or change the GLFW_DIR, by giving it as parameter to the make command for example).

GLFW didn't compile properly for me in cygwin with the default settings though, I had to use cmake . -DWIN32=1 -DGLFW_BUILD_TESTS=0 -DGLFW_BUILD_DOCS=0 -DGLFW_BUILD_EXAMPLES=0

The emulator currently needs Opengl 3.3 support, but it could quite easily be backported to opengl 2.

ec1oud commented 8 years ago

So the head of the led branch is now

afd5be1 Update visualizer

With that one, I don't see any LED animation, they are staying off. But I do see the LCD image toggling back and forth erratically. Sometimes much faster than usual.

fredizzimo commented 8 years ago

Hm, that's strange. I must have broken something when doing the emulator stuff.

I will check it, after I have made blitting support for this https://github.com/fredizzimo/infinity_ergodox/pull/11

ec1oud commented 8 years ago

But https://youtu.be/ocLzjHaLDtg is from ba6b411, seems OK.

Git bisect says afd5be1 is the first bad commit. Which doesn't make sense to me.

fredizzimo commented 8 years ago

Some quick update.

I have not completely found out what's wrong. But I found a few problems.

  1. The erratic LCD image toggling is caused by the system somehow waiting for much longer than it should. It happens at the start of the LED animation cycle, where the LEDs are supposed to stay on for 1 second. At this time there's no fading going on, so the system can wait. It's supposed to wait for around 0.5 seconds (verified by prints), but it actually waits for 5 - 20 seconds. This also causes the following animations to toggle too fast, since the animation tries to catch up. This behavior is probably caused by the usage of the ugfx api https://github.com/fredizzimo/tmk_visualizer/commit/d79e94adb1182ae867df0cc7621ef3d44d213bbc.
  2. It seems to send the right values to the LEDs, so I don't undertand if the LEDs don't turn on at all. It also seems to be completely unrelated to the changes.
  3. Each update step is taking much longer than supposed, and it's not related to the cos functions. Calculating those takes less than 1 ms. But the whole update, probably caused by either the LED or LCD transfer to the actual hardware takes over 200ms, which means that we can only get 5 FPS for the animations, so that's very bad. I need to check this further.

The difference between f7cedcdfaf9e44eab9fe3976c451625feda50796, which is working(?), and afd5be1d81348ccf1974e2c91d3614e7d809e5db, is the update of the tmk_visualizer submodule. They both actually points to the same commit message, but I amended a commit together, so they are actualy two changes. The only difference is that the following line in led_test.c was changed from

float normalized_index = 1.0f - (index / (num - 1) * two_pi);

to

 float normalized_index = (1.0f - index / (num - 1)) * two_pi;

Are you sure you ran git submodule update --recursive for each bisect test? Otherwise the tmk_visualizer submodule might not have been at the correct change.

Edit: I think the animation now looks otherwise correct, expect that the bug that was in the amended commit bug is there. This causes the sine gradients to start from around the second key, rather than the first one.

ec1oud commented 8 years ago

I was doing the bisect build/test steps like this

git submodule sync --recursive; git submodule update --init --recursive; git clean -dfx; make && sudo make program

fredizzimo commented 8 years ago

Hi @ec1oud,

I have now fixed the problem, at least with the erratic LCD image toggling. There was a bug in either the ugfx library documentation or implementation, which says that the function takes a system tick parameter, but in reality it should be milliseconds. I will report that bug to them at some point.

Also due to an integer overflow caused by this, the behavior was not consistent, which could explain why it appeared to be working for some commits. BTW, I can see the problem in the video you posted. At the end of it, the layer bitmap stays on for around 5 seconds, and after that it flickers a bit.

I think that the missing LED animation could have been caused by the same issue, but I'm not sure.

r2d2rogers commented 8 years ago

I'm eager to try this, would it still be useful to have reports from the LED branch here, or should I wait for the QMK side?

fredizzimo commented 8 years ago

Thanks @r2d2rogers, yes it's still useful to have it tested here. The code that I have done here will be almost directly ported to QMK, so it would be nice confirm that everything works before that.

r2d2rogers commented 8 years ago

I was able to flash and see the led pattern progression. I'll get setup to post a YouTube video tonight so you can see if it's doing exactly what you expect.

r2d2rogers commented 8 years ago

Here is the video I promised. https://youtu.be/GXt1RlNkzQo

fredizzimo commented 8 years ago

Hi, @r2d2rogers

Thanks for the video. Unfortunately it showed the same problem as @ec1oud saw, and that I already fixed. So I checked the history, and it turned out that I had forgot to push the LED branch with the fixes to Github, I had only pushed the fixes in the visualizer submodule.

I just pushed it, so now it should hopefully work better.

r2d2rogers commented 8 years ago

Another video with the latest changes, let me know when you'd like me to use the qmk version. -- Link removed, I think I missed the submodule update, reflashing and will post a new video.

r2d2rogers commented 8 years ago

Now I don't get key led's lit at all, but the LCD responds much faster for layer changes.