search

DustinWatts / FreeTouchDeck

For interfacing with Windows/macOS/Linux using an ESP32, a touchscreen and BLE.
https://www.youtube.com/dustinwatts
MIT License
624 stars 126 forks source link

Flexible configuration #51

Closed sle118 closed 3 years ago

sle118 commented 3 years ago

This is just a comment. @DustinWatts and the other contributors to this project. Feel free to close it.

I wanted to thank you for taking the time to build this. It was exactly what I was looking for!

I took the time to port the code to the TTGO T-Watch 2020 v1 which I happen to own. This wasn't a huge effort; the watch has the same capacitive touch controller that the project uses. A couple of notable differences are the screen resolution (smaller) as well as the onboard power management chip which needs to be told to power on in order to get the back light to work.

Just for fun, I spent a couple of hours refactoring a bit of the code, especially around how menus/screens are represented and loaded internally. I've done it in a way that would allow any number of menus to be created, each with an arbitrary number of buttons, and an arbitrary number of actions, all the while preserving the JSON format which you are currently using (for the sake of staying compatible with existing installs).

Essentially, there are now a few more classes:

Let me know your thoughts and if this is something which you'd like to see some day.

danielhunt commented 3 years ago

Amazing work @sle118 👏

sle118 commented 3 years ago

@danielhunt you haven't seen a line of code yet... shall you really trust that it's done ?? ;)

danielhunt commented 3 years ago

Ah good point. That sounds like amazing work so 🤣👏

sle118 commented 3 years ago

Perhaps someone can enlighten me here. If I press LEFT_SHIFT, then type some characters, then release the shift, it seems as though the first subsequent character is always being "shifted up".

What am I missing?

DustinWatts commented 3 years ago

Can you give an example of the key sequence you are using? Maybe I can figure out what is happening. The blekeyboard.Write() function makes a distinction between upper and lower cases.

sle118 commented 3 years ago

The action is defined as 

["This is some text{TAB}After tab key {LEFT_SHIFT}some capitalized text",
 ". And this text has the shift key released automatically. {RETURN}",
"{LEFT_SHIFT}some more capitalized text{RELEASEALL}. Followed by a manual release"]

and this produces the following output

This is some text   After tab key SOME CAPITALIZED TEXT> And this text has the shif key released automatically. 
SOME MORE CAPITALIZED TEXT> Followed by a manual release

As you can see, it seems like the character that comes right after the release . is interpreted as > which is the equivalent of {LEFT_SHIFT}..

I'm starting to think that we need to listen for a confirmation that the keyboard event was properly dispatched before sending the next char. It might be possible to wait for an arbitrary amount of time, but this would be very much trial and error to fine tune?

sle118 commented 3 years ago

@danielhunt now you can see for yourself! I pushed the changes to my repo

sle118 commented 3 years ago

I have started experimenting with SD cards. I'm thinking of a seamless integration here... insert a SD card with the data folder structure and you're up and running. Otherwise, the system falls back on SPIFFS.

seems simple enough to achieve, except for the "get free space" for which total space and used space methods aren't standard in the fs::FS class.

to be continued, but this is likely the best way to configure the device, short of having the UI.

Can anyone confirm if, assuming you've selected log level=3, the drawing seems snappier or not with my fork when compared to the original branch? I'm curious to see if moving rendering to IRAM made a difference. I think it is a tiny bit faster, but I am biased.

Being able to leverage the SD card means we could improve on performance. For example, images could be converted on the fly to RGB565 (16 bits), which is the native TFT display driver's bit depth, from the current 32 bits. Today, the conversion is done line by line, which limits the buffer transfer speed from spiffs to the display. The SD card would transparently hold a mirror 16 bits image of each icon folder image. And honestly, even a 8gb card is really cheap nowadays, which extends the menu capabilities to a great deal. It would be possible to bundle the entire material icons library so it is available to users. Revising menus could be done in notepad++ on the SD card.

sle118 commented 3 years ago

@coryb now would be a good time to have a look at my fork. I think I've reached a state where refactoring of the code will be limited to peripheral features. The core is streamlined sufficiently good now, and offers a good base for extensibility.

I've moved so far in the code that I haven't documented much, though

briight commented 3 years ago

@sle118 hey could you enable issues section on your fork so i can start adding things i find =) thank you =D

sle118 commented 3 years ago

@briight for now you could list them here. I intend to send a pull request towards Dustin's repo, probably on the development branch or a new one given how far the code base has gone compared to the original.

Note that there is currently no support for for the configurator. Everything needs to be done through either flashing files from the data folder, or by using the console through the serial port with a terminal

briight commented 3 years ago

Ah okies kool

Is there anyway to build the config for my menus with a gui or it all manual atm ?

briight commented 3 years ago

Screen Rotation setting dosnt work on esp32 dev

sle118 commented 3 years ago

Ah okies kool

Is there anyway to build the config for my menus with a gui or it all manual atm ?

It is an manual, but please have a look at the examples I provided; you will see how simple it has become.

sle118 commented 3 years ago

Screen Rotation setting dosnt work on esp32 dev

Could you please provide more details? Running on the esp32-touchdown works for me and I don't have many more hardware platforms to test against

briight commented 3 years ago

I change the number next to screen rotation and it dosnt change anything. Im assuming it like the old one 1,2,3,4 and the 90⁰ diffrent

sle118 commented 3 years ago

Please try changing the value in the config json file, either through showconfig/setconfig or via data folder upload

sle118 commented 3 years ago

Behold everyone. I've added support for an SD card which will seamlessly use whichever storage class that's available. If other users have different storage type, they will have the ability to expand easily.

While doing this, however, I went ahead and did one more refactoring of the code. This time, it isn't changing the code itself; I just decluttered the main FreeTouchDeck.ino file so it is less intimidating with stuff that is really low level. These pairs of .cpp and .h file were created, or the .h file was split into a proper .cpp and .h file

briight commented 3 years ago

Please try changing the value in the config json file, either through showconfig/setconfig or via data folder upload

changed in general.json worked on the rotation ty

sle118 commented 3 years ago

ok. I'll make sure the system pulls from the build on the initial start, rather than from the json file itself. After initial start is complete, the json will be updated with the rotation value defined in the build.

briight commented 3 years ago

I think it might be good like it is. If u want to change the oriantation I just have to edit a file on ur SD card if u are using it

sle118 commented 3 years ago

I think it might be good like it is.

It could work, except that when you build a certain target (e.g. with some user configurations done), you'd expect that you'd boot with the right orientation after flashing. It's a minor change, really

briight commented 3 years ago

That's what I assumed

DustinWatts commented 3 years ago

Can anyone confirm if, assuming you've selected log level=3, the drawing seems snappier or not with my fork when compared to the original branch? I'm curious to see if moving rendering to IRAM made a difference. I think it is a tiny bit faster, but I am biased.

I have the same feeling. I did a side by side comparison with the original code and it is faster. Not by much, but enough to make a huge difference in how it feels!

DustinWatts commented 3 years ago

to be continued, but this is likely the best way to configure the device, short of having the UI.

I love this option. I'm pretty used to re-uploading the SPIFFS each time, but it would be a huge bonus for users!

re the UI (Configurator I mean), someone on my Discord is making some nice progress with a mockup of a new configurator...

DustinWatts commented 3 years ago

I noticed that keydelay is not used between characters in FREETEXT, only between actions. Adding a small delay when typing text can make a huge difference in characters not being missed. So my idea is to also add the delay between each character. Best practice I found is to iterate through a char array with a small delay (10ms) between each char...

sle118 commented 3 years ago

to be continued, but this is likely the best way to configure the device, short of having the UI.

I love this option. I'm pretty used to re-uploading the SPIFFS each time, but it would be a huge bonus for users!

re the UI (Configurator I mean), someone on my Discord is making some nice progress with a mockup of a new configurator...

I think we should coordinate on that configurator. There are a few things that the backend can do to ensure consistency with the UI. For example, the backend should

In other words, the UI should never make any assumption as to what the backend supports, with the exception of basic LOCAL action like {MENU:menuname}, which the frontend should be able to parse and validate.

sle118 commented 3 years ago

I noticed that keydelay is not used between characters in FREETEXT, only between actions. Adding a small delay when typing text can make a huge difference in characters not being missed. So my idea is to also add the delay between each character. Best practice I found is to iterate through a char array with a small delay (10ms) between each char...

ok. I'll make that a config option, alongside the current delay, which is used between actions

sle118 commented 3 years ago

I have the SD/SPIFFS storage implemented (had to rewrite the whole class because my editor froze and an empty file was written to disk).

There's an annoying memory hog which I need to investigate. I'm not sure at this point if it is caused by initializing the SD card or if it is something else.

briight commented 3 years ago

i got around to setting up one of my pages iv noticed that some keys stick on (so far only used {RIGHT_CTRL}{RIGHT_SHIFT}) causing docs to be highlighted and zoomed in lol, like a release key function isnt there or not working. other things i carnt find are delay on key presses, i understand there are more important things going on =) and i carnt get my head around the latching,

i understand that some thing wont have made it into it and are not priority

great work dude

sle118 commented 3 years ago

delay is a local action: {DELAY:200} Release happens automatically at the end of each line in the actions array. Release can also be forced with {RELEASEALL}

you can check in UserActions to find out the valid actions that are built in

I just checked in some WIP progress on SD card support

sle118 commented 3 years ago

Just saying.... Would there be any interest in being able to use JavaScript to do some fancy actions when pressing buttons? I'm referring to a diminutive version of the language, but nevertheless, any interest?

DustinWatts commented 3 years ago

Just saying.... Would there be any interest in being able to use JavaScript to do some fancy actions when pressing buttons? I'm referring to a diminutive version of the language, but nevertheless, any interest?

You might want to have a look at https://github.com/DustinWatts/FreeTouchDeck-Helper I wrote this app in Electron/NodeJs to give the user access to the shell using the "helpers" (technically any shortcut). You mean something like that?

DustinWatts commented 3 years ago

The FREETEXT "This is some text{TAB}After tab key{LEFT_SHIFT}some capitalized text", ". And this text has the shift key released automatically. {RETURN}", "{LEFT_SHIFT}some more capitalized text{RELEASEALL}. Followed by a manual release" didn't work as expected. Like you said earlier a . became an >. I also noticed that ALT+TAB was holing down ALT. So I rewrote that bit and it now works as expected:

line 326:

else
{
 for (auto key : values)
 {
  bleKeyboard.write(key);
  delay(generalconfig.keyDelay);
 }
 bleKeyboard.releaseAll();
}

This

This is some text   After tab keySOME CAPITALIZED TEXT> And this text has the shift key released automatically. 
SOME MORE CAPITALIZED TEXT> Followed by a manual release

has become this:

This is some text   After tab key. SOME CAPITALIZED TEXT. And this text has the shift key released automatically. 
SOME MORE CAPITALIZED TEXT. Followed by a manual release
sle118 commented 3 years ago

Nope. I was referring to running a subset of Java ON the touchdown itself. If is possible to expose custom objects/methods to allow gpio manipulations, calling local actions, etc.

The only thing I'm not too certain about is if there's enough memory left for this. We might want to consider looking at the biggest memory hitters and reduce their footprint wherever possible. Xtaskcreate with unnecessary high stack allocation for example.

But perhaps this would be a better idea for an esp32-touchdown+ beefed up with 4mb psram ;)

sle118 commented 3 years ago

line 326:

else
{
 for (auto key : values)
 {
  bleKeyboard.write(key);
  delay(generalconfig.keyDelay);
 }
 bleKeyboard.releaseAll();
}

What do you have as the key delay in your config? I think you might have made another change somewhere else? This bit of code happens to run on keys like shift, ctrl, etc. Releasing at this point of time effectively means the system will press, say, LEFT_SHIFT then release it before sending the following key sequences. This is because the ActionSequence parser breaks a single action like into multiple discrete actions. So this action: This is some text{TAB}After tab key{LEFT_SHIFT}some capitalized text

Turns out to be split into 5 different actions.

The parser keeps track of keys requiring release during the parsing

                    auto lastAction=Actions.back();
                    if(lastAction->NeedsRelease)
                    {
                        releaseKeyList.insert(releaseKeyList.end(), lastAction->Values.begin(), lastAction->Values.end()); 
                    }

and appends a release action at the end of the sequence whenever needed

        if(releaseKeyList.size()>0)
        {
           releaseParameters.push_back("RELEASEKEY");
           auto releaseAction = new FTAction(releaseParameters);
           releaseAction->Values.insert(releaseAction->Values.end(), releaseKeyList.begin(),releaseKeyList.end());
           Actions.push_back(releaseAction);
        }

So when the action sequence class plays all the actions, it eventually reaches the last one and performs the release

 {"RELEASEKEY", [](FTAction *action)
         {
             uint16_t forcedDelay = max(generalconfig.keyDelay, (uint16_t)200); // force a 300ms delay
             LOC_LOGD(module, "Releasing %d keys", action->Values.size());
             for (auto k : action->Values)
             {
                 delay(forcedDelay);
                 bleKeyboard.release(k);
             }
             return true;
         }},

I think this is the actual faulty code. It seems that calling the release at this point just "lines up" the key for being released when the subsequent key presses are done. I think I need this is where I need to call a single releaseAll. Something like:

{"RELEASEKEY", [](FTAction *action)
         {
             uint16_t forcedDelay = max(generalconfig.keyDelay, (uint16_t)200); // force a 300ms delay
             LOC_LOGD(module, "Releasing %d keys", action->Values.size());
            //  for (auto k : action->Values)
            //  {
            //      delay(forcedDelay);
            //      bleKeyboard.release(k);
            //  }
            bleKeyboard.releaseAll();
             return true;
         }},

EDIT: I tried changing the code as shown above, still no dice. I even tried a 400ms delay... Log shows the action happened

[D][FTAction.cpp:299] Execute(): Executing Action: Type: LOCAL, value:  RELEASEALL
[D][FTAction.cpp:190] GetParameter(): Getting parameter 0 a total of 1 entries
[D][FTAction.cpp:198] GetParameter(): Found entry 0 : RELEASEALL
[D][FTAction.cpp:465] CallActionCallback(): Found the callback in the map
[D][FTAction.cpp:482] CallActionCallback(): Calling function
[D][UserActions.h:201] operator()(): Releasing All
[D][BLECharacteristic.cpp:786] onNotify(): BLECharacteristicCallbacks
[D][BLECharacteristic.cpp:787] onNotify(): BLECharacteristicCallbacks
[D][BLECharacteristic.cpp:798] onStatus(): BLECharacteristicCallbacks
[D][BLEDevice.cpp:102] gattServerEventHandler(): gattServerEventHandler [esp_gatt_if: 4] ... Unknown
[D][BLECharacteristic.cpp:799] onStatus(): BLECharacteristicCallbacks

Is this a timing problem? should the system wait for a full second or something?

sle118 commented 3 years ago

Is there a way to register a callback for when the ble keyboard is done communicating the keys in its queue? I think this might be the main culprit here and I'd like to try waiting for the buffer to empty before releasing.

briight commented 3 years ago

tried ur latest fix but carnt put it on my esp32

'touch_calibrate' was not declared in this scope'

using

define ESP32DEVKIT

sle118 commented 3 years ago

touch_calibrate

Do you have a capacitive touch screen? If so, make sure you define this #define USECAPTOUCH

briight commented 3 years ago

Do you have a capacitive touch screen?

no i dont

when i add this #define USECAPTOUCH my touch stops working

when its not there it just throws up that error in program

im using the devkit and looks like even though #define USECAPTOUCH isnt needed its looking for it to use

sle118 commented 3 years ago

Ok. I'll have a look later today; it is definitely something that can be resolved easily. Since my main development platform is now the touchdown, I tend to forget making sure other platforms work as well

briight commented 3 years ago

the fix worked thank you

keys are still sticking on

sle118 commented 3 years ago

the fix worked thank you

keys are still sticking on

Yep. I am giving that issue some TLC now

edit: I found the culprit; keyboard and local actions each have their own execution queue. What was happening is that the system was queuing keyboard actions to the low priority actions queue, and the release all to the higher priority queue (same as local actions. The end result was that keys were released before they were pressed!

I'm going to push the fix out. Here are a few other changes

edit: not pushed yet... adding bitmap drawing improvement...

sle118 commented 3 years ago

changes are pushed now. Seeking feedback!

sle118 commented 3 years ago

ahhh oups. Just pushed a small one. Actions were stuck. Hoping no one tried volume up with that previous push :S

danielhunt commented 3 years ago

I did. It was ... an experience :D

briight commented 3 years ago

changes are pushed now. Seeking feedback!

Works great no sitcky keys

could you explain how i would add a delay to a key press as obs need one to pick up on hotkeys thanks

sle118 commented 3 years ago

changes are pushed now. Seeking feedback!

Works great no sitcky keys

could you explain how i would add a delay to a key press as obs need one to pick up on hotkeys thanks

Just insert {DELAY:100} where you need the delay, and set value as needed.

If you share some of your obs preset, I'll include them in the build.

sle118 commented 3 years ago

I did. It was ... an experience :D

I was lucky enough that I pushed mute. And I was like... Why the heck is sound stuttering?? 8)

First rule is "don't test with audio on the first attempt"

briight commented 3 years ago

Just insert {DELAY:100} where you need the delay, and set value as needed.

i added this it just delays the key press. not delays the release of the keys