search

DustinWatts / FreeTouchDeck

For interfacing with Windows/macOS/Linux using an ESP32, a touchscreen and BLE.
https://www.youtube.com/dustinwatts
MIT License
632 stars 128 forks source link

Flexible configuration #51

Closed sle118 closed 3 years ago

sle118 commented 3 years ago

This is just a comment. @DustinWatts and the other contributors to this project. Feel free to close it.

I wanted to thank you for taking the time to build this. It was exactly what I was looking for!

I took the time to port the code to the TTGO T-Watch 2020 v1 which I happen to own. This wasn't a huge effort; the watch has the same capacitive touch controller that the project uses. A couple of notable differences are the screen resolution (smaller) as well as the onboard power management chip which needs to be told to power on in order to get the back light to work.

Just for fun, I spent a couple of hours refactoring a bit of the code, especially around how menus/screens are represented and loaded internally. I've done it in a way that would allow any number of menus to be created, each with an arbitrary number of buttons, and an arbitrary number of actions, all the while preserving the JSON format which you are currently using (for the sake of staying compatible with existing installs).

Essentially, there are now a few more classes:

Let me know your thoughts and if this is something which you'd like to see some day.

sle118 commented 3 years ago

Just insert {DELAY:100} where you need the delay, and set value as needed.

i added this it just delays the key press. not delays the release of the keys

I think I know why this is happening and I'm thinking I need to have all local actions run in the same queue as the keyboard. Stay tuned, I'll look it up.

In the meantime, could you share your obs menu here?

briight commented 3 years ago 
[
   {
    "type":"HOME",
    "name":"OBS",
    "colscount":3,
    "rowscount":4,
    "logo":"SCENESWOW.bmp",
    "buttons":[
    {
        "logo":"HIDVID.bmp",
        "latchedlogo":"HIDVID.bmp",
        "type":"LATCH",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F17}"
        ]
      },
      {
        "logo":"volumedown.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F13}"
        ]
      },
          {
        "logo":"volumeup.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F14}"
        ]
      },
      {
        "logo":"play.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F15}"
        ]
      },
      {
        "logo":"skip.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F16}"
        ]
      },
          {
        "logo":"START.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F17}"
        ]
      },
          {
        "logo":"BRB.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F18}"
        ]
      },
          {
        "logo":"GAME.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F19}"
        ]
      },
          {
        "logo":"END.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F20}"
        ]
      },
      {
        "logo":"headsetmic.bmp",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F21}"
        ]
      },
      {
        "logo":"discord.bmp",
        "type":"LATCH",
        "actions":[
          "{RIGHT_CTRL}{RIGHT_SHIFT}{F22}"
        ]
      }
    ]
  },
  {
    "type":"HOME",
    "name":"Media",
    "logo":"music.bmp",
    "buttons":[
      {
        "logo":"mute.bmp",
        "type":"LATCH",
        "actions":[
          "{MUTE}"
        ]
      },
      {
        "logo":"volumedown.bmp",
        "actions":[
          "{VOLUME_DOWN}"
        ]
      },
      {
        "logo":"volumeup.bmp",
        "actions":[
          "{VOLUME_UP}"
        ]
      },
      {
        "logo":"play.bmp",
        "actions":[
          "{PLAY_PAUSE}"
        ]
      },
      {
        "logo":"prev.bmp",
        "actions":[
          "{PREVIOUS_TRACK}"
        ]
      },
      {
        "logo":"skip.bmp",
        "actions":[
          "{NEXT_TRACK}"                                
        ]
      }
    ]
  }
]

im testing with

{ "logo":"discord.bmp", "type":"LATCH", "actions":[ "{RIGHT_CTRL}{RIGHT_SHIFT}{F22}" ] } atm

sle118 commented 3 years ago

if your icons are free of charge, could you please share them somewhere as well? for example on imgur.com or somewhere else

sle118 commented 3 years ago

@briight please try with the latest changes, and make sure you try the following sequence

{
"logo":"discord.bmp",
"type":"LATCH",
"actions":[
  "{RIGHT_CTRL}{RIGHT_SHIFT}{F22}{DELAY:200}"
]
}
briight commented 3 years ago

it dosnt work it like delays the press so it happens with a 200ms delay press - 200ms - delay - action - relase im after press - hold action 200ms - relase

sle118 commented 3 years ago

yea it like delays the press so it happens with a 200ms delay

does it mean it works?

Could you describe the exact sequence (press, ... delay, release, ... ) that would need to work?

briight commented 3 years ago

im after press - hold action 200ms - relase

sle118 commented 3 years ago

you mean holding F22?

I'm confused with the use of short acronyms.

briight commented 3 years ago

obs dosnt pick up on the quick combo presses iv had to add delays into them on the old FTD 110214018-8b2d5780-7e9a-11eb-83bb-cfbfa18cad62

my hot keys on obs are (right control + right sift + f22) and a delay of 100ms so obs acutaly see them as it dosnt pick them up if obs is not in focus

sle118 commented 3 years ago

I checked the old code and it definitely seems like F22 is being held during that delay as well.

So I'm going to make something that will allow something like

{
"logo":"discord.bmp",
"type":"LATCH",
"actions":[
  "{RIGHT_CTRL}{RIGHT_SHIFT}{F22:200}"
]
}

This will allow inserting a delay between the press and the release

briight commented 3 years ago

{RIGHT_CTRL}{RIGHT_SHIFT}{F22} held down all at once for 100ms to clarify

I'll upload my icons when I have chance

sle118 commented 3 years ago

yes, this is expected. The playback of {RIGHT_CTRL}{RIGHT_SHIFT}{F22:200}

will be as follow: **press RIGHT_CTRL, press RIGHT_SHIFT, press F22, delay 200ms, release F22, release all**. The old logic would have done press RIGHT_CTRL, press RIGHT_SHIFT, press F22, release F22, delay 200, release all, which obviously was wrong.

edit: changed pushed

DustinWatts commented 3 years ago

I checked the old code and it definitely seems like F22 is being held during that delay as well.

So I'm going to make something that will allow something like

{
"logo":"discord.bmp",
"type":"LATCH",
"actions":[
  "{RIGHT_CTRL}{RIGHT_SHIFT}{F22:200}"
]
}

This will allow inserting a delay between the press and the release

This is actually done because of the way OBS polls the keyboard. Especially when under heavy load, it stops catching keystrokes.

I'm taking a look at the SD card support atm. Is the expected behavior that once loaded from SD card, the SD card needs to remain in place? I mean, when the SD card is removed, it will default back to something? Or should the SD card config be copied? I think this might cause an issue with logos (free space wise), but menu and general config could be copied.

Thinking out loud... a button in the settings menu, letting users copy the SD card config to SPIFFS and thus making it permanent would be a great feature.

Also, could we display a message on the screen when it starts loading from SD? I inserted an SD card and didn't realize that it was loading. I had a black screen and decided to power cycle because I thought there was a boot issue. After that, boot loop. Only then I looked at the serial monitor. I think most users will not have the serial monitor open so might do the same...

DustinWatts commented 3 years ago

Something else I just found. I can select multiple menus, and it will load them one after another. The back button then takes me to all other menus that I touched, and eventually back where I came from (in my case the main menu). But the same behavior is observed with function buttons...

sle118 commented 3 years ago

Thinking out loud... a button in the settings menu, letting users copy the SD card config to SPIFFS and thus making it permanent would be a great feature.

I think it's a good idea. However, I'd restrict the copy to only the json files and not the logos because of space constraints.

Also, could we display a message on the screen when it starts loading from SD? I inserted an SD card and didn't realize that it was loading. I had a black screen and decided to power cycle because I thought there was a boot issue. After that, boot loop. Only then I looked at the serial monitor. I think most users will not have the serial monitor open so might do the same...

Good point. Noted. I was also thinking of reserving a dedicated screen area for status like battery, wifi, etc...

But the same behavior is observed with function buttons...

Could you be more specific?

Also, if you have another device with your own firmware on it, could you try to compare the drawing speed between both? I'm curious if using RAM and a larger read buffer helps or not. I've modified the BMP drawing allocate a larger buffer on the heap rather than stack and to pull more lines simultaneously.

The latest parser now allows for {TAB:200}{F12:100} for arbitrary delays between press and release. Or (this wasn't tested but should work), {a:100} to press, hold 100ms and release key a or any other valid character/key.

edit: Also coming up next is the proper boot logo drawing... finally!

DustinWatts commented 3 years ago

Could you be more specific?

Absolutely. I can drag my finger across the buttons and it will register all them in sequence. When I let go, all actions (or loading a menu) are preformed. Kinda like the Android screen unlock

Also, if you have another device with your own firmware on it, could you try to compare the drawing speed between both? I'm curious if using RAM and a larger read buffer helps or not. I've modified the BMP drawing allocate a larger buffer on the heap rather than stack and to pull more lines simultaneously.

I did that the last time I commented about this. That was definitely faster. However the recent push seems to have made it a bit slower again.

sle118 commented 3 years ago

Could you be more specific?

Absolutely. I can drag my finger across the buttons and it will register all them in sequence. When I let go, all actions (or loading a menu) are preformed. Kinda like the Android screen unlock

Ok. This is a side effect of executing the actions upon release; I started this to allow for users to slide fingers away from an accidentally pressed button ;)

I did that the last time I commented about this. That was definitely faster. However the recent push seems to have made it a bit slower again.

This is surprising to say the least. One thing that might give this impression is that the button matrix (shapes and labels if no logo) is drawn first, then images 2nd. Please try something like a 6x6 menu, all with logos and see if the total drawing time is better or not. If it isn't, then I'll need to check for bottlenecks.

briight commented 3 years ago

just tried the push u did about 2 hours ago loading image fixed dragging finger on buttons fixed delay working and obs responding out of focus
Brilliant dude ty

sle118 commented 3 years ago

@briight if you have an external SD card like on the touchdown, you can experiment!

sle118 commented 3 years ago

I'm looking to try using TJpg_Decoder to use JPG files instead of BMP ones (the /data/logo directory goes from 952k to 108k without looking at additional compression. This makes a material difference to the data directory size 😱 )

@sle118 your abstraction of the BMP classes is 👌 but bmp as a concept is still littered through the codebase (default images and like) Would it be worth abstracting this away a bit more so that a JPGImage class could be easily swapped into place instead of BMPImage?

So I just started refactoring the image wrapper class and if this was a work project, I'd go for some dependency injection pattern or for a form of inheritance, but I'm not sure I want to go there. I hate to say, though, that I really don't like scaling code with case statements so I just might bite the bullet and do the proper thing. JPG is good, but then there's PNG and SVG as well (there's one SVG parser around, but my first attempt at compile wasn't successful).

One of the things I did to reduce the overall space needed was to allow using labels instead of icons. It may not be as nice, but it draws much faster and takes very little room

briight commented 3 years ago

if your icons are free of charge, could you please share them somewhere as well? for example on imgur.com or somewhere else

here are my icons i use https://imgur.com/a/ls8wjAN

i will hook up my sdcard today and try and find an old sdcard

briight commented 3 years ago

hooked up the sd card carnt get it to work also not getting any info back on the serial monitor to see why

edit i got info on my serial monitor

also got my sdcard to read on diffrent examples

sle118 commented 3 years ago

also not getting any info back on the serial monitor to see why

could you uncomment the following line in the sketch and recompile? #define DEFAULT_LOG_LEVEL LogLevels::DEBUG

and make sure the board configuration has core log level set to debug.

Also call command loglevels 4 to set all log level to DEBUG, then restart

I have yet to review the info log level and make sure messages make sense without cluttering the serial connection

briight commented 3 years ago

ok done that got a long file now
just looking through it

[I][FreeTouchDeck.ino:155] setup(): Starting system. [I][Storage.cpp:125] InitFileSystem(): Initializing storage SPIFFS [I][Storage.cpp:135] InitFileSystem(): Using File system SPIFFS. [I][Storage.cpp:120] InitFileSystem(): Reached the end of the file systems list [I][ConfigLoad.cpp:265] loadConfig(): Loading configuration from file name: /config/general.json

is the only storage i can find

sle118 commented 3 years ago

Ah!

Ok. You have to specify the GPIO for the SD card reader. For the TouchDown, it is #define SDDAT3 25 in the UserConfig.h file.

Could you remind me which board you are using?

briight commented 3 years ago

DOIT ESP32 DEVKIT V1 Board u know how long i was looking for #define SDDAT3 25 using the search and didnt find it =(

didnt work is there anywhere else i need to define the GPIO the the sd card?

DustinWatts commented 3 years ago

didnt work is there anywhere else i need to define the GPIO the the sd card?

In UserConfig.h try adding #define SSDAT3 26 under:

#elif defined(ESP32DEVKIT)
#define SCREEN_ROTATION 1
#define FLIP_TOUCH_AXIS
#define INVERSE_Y_TOUCH
#define touchInterruptPin GPIO_NUM_27

So it becomes:

#elif defined(ESP32DEVKIT)
#define SCREEN_ROTATION 1
#define FLIP_TOUCH_AXIS
#define INVERSE_Y_TOUCH
#define touchInterruptPin GPIO_NUM_27
#define SSDAT3 26

Save and reupload the code...

briight commented 3 years ago

i just found this already testing =P

briight commented 3 years ago

that why i couldnt find it wasnt in the board section i am using

DustinWatts commented 3 years ago

Could you remind me which board you are using?

The ESP32 DevKitC and TFT combiner board I made uses some slightly different pins. On that board SSDAT3 is on GPIO26.

briight commented 3 years ago

got the sd card working now

images load abit slow from menu to menu about 2 seconds for all to be loaded 3x4

sle118 commented 3 years ago

well, I decided to goo full blown on polymorphism for the image handlers, but I have to say this sort of things is easier to do in c# than in c++, especially because my advanced c++ knowledge is rather rusty. So jpg will take a bit of time!

briight commented 3 years ago

so i added another menu to my sdcard almost the same as my first one with a few keys changed same icons the only icon changed was the one on the main screen it loads up the menus on the main screen fine, once in one of the menus the buttons are draw but no images and has to be restarted in the serial monitor i get

[E][ImageWrapper.cpp:346] Draw(): Error allocating 12540 bytes of buffer for image drawing!

so i tryed uploading it straght to the eps32 and its fine

sle118 commented 3 years ago

so i added another menu to my sdcard almost the same as my first one with a few keys changed same icons the only icon changed was the one on the main screen it loads up the menus on the main screen fine, once in one of the menus the buttons are draw but no images and has to be restarted in the serial monitor i get

[E][ImageWrapper.cpp:346] Draw(): Error allocating 12540 bytes of buffer for image drawing!

so i tryed uploading it straght to the eps32 and its fine

Have a look at the UserConfig.h file. There you will see the following line #define BITMAP_BUFFER_FREE_RAM_PCT 0.30. this is the percentage of memory that is allowed to be allocated for image drawing buffering. I might have been too aggressive here. Please try 0.15 and see if it helps.

The fact that it works from the internal storage is likely because reading from the SD card takes some additional resources.

sle118 commented 3 years ago

Could you remind me which board you are using?

The ESP32 DevKitC and TFT combiner board I made uses some slightly different pins. On that board SSDAT3 is on GPIO26.

Perhaps this should be called out specifically in the UserConfig.h file as #ifdef TFTCOMBINER or something along these lines to make it a separate "preconfigured" section?

briight commented 3 years ago

i dont use a tft combiner , i also have other thing connected volume sliders on gpio 39,34,35,32,33,25

sle118 commented 3 years ago

i dont use a tft combiner , i also have other thing connected volume sliders on gpio 39,34,35,32,33,25

Neat! Did you go and implement AVRCP to get absolute volume controls? How do you keep the sliders levels in sync with the host's volume levels?

briight commented 3 years ago

i put another project called deej into ftd send analog data over bluetooth to deej program that hooks into the windows volume thingy carnt remember what its called now

sle118 commented 3 years ago

I just wanted to let everyone know that JPG files are now supported and will be the de-facto format going forward. I did replace all built in icons with JPG files, so you will need to adjust your personal menus.

Also, SD card init procedure will now show progress on the TFT screen.

I'll push changes when I have done sufficient testing

sle118 commented 3 years ago

I just pushed the changes. JPG is now supported FROM SPIFFS (internal storage). Don't try on SD card yet. SD support needs some attention and I'm not sure if I'm having issues because of a faulty card or because of the code. So for now stay on SPIFFS ;)

sle118 commented 3 years ago

for anyone still following this story, I have just pushed what looks like a somewhat good starting point for SD card support INCLUDING JPG !

Using JPG as opposed to using bmp means there's less data to transfer during drawing of screens, which somewhat compensates for the fact that SD cards in general are slower.

I still need to figure out a way to gracefully handle the "unexpected removal" of the card during use, but that'll come!

Hoping you all enjoy the added simplicity of editing the configuration straight from the SD card!

Comments are welcomed

Edit: I spoke too soon. I'm still getting a crash and I'm not sure why. On the regular esp-idf, I would enable core dump to figure out what's wrong, but I doubt this is an option on the Arduino platform.

sle118 commented 3 years ago

I pushed another set of changes today. These are mostly geared towards getting more stability out of the SD card. I did identify one leak and I'm hoping fixing it will have helped a bit.

Worst case, there is another memory saving I can do, which is to not initialize the spiffs when SD card is detected

sle118 commented 3 years ago

So for anyone not following on discord, I just pushed some changes

To reduce memory footprint, I also opened a pull request https://github.com/DustinWatts/ESP32-BLE-Keyboard/pull/2

sle118 commented 3 years ago

And by the way, I'm very close to calling it finish. There's a few renames of variables and methods to do, but overall I'm happy with the current state as I think it has reached a stable enough state.

Next I plan on experimenting with the Arduino cli to build for various targets using the GitHub workflows. Being able to pass extra defines on the command line means even the TFT library can be configured without having to mess with include files. I've built something similar on a different repository, except not for Arduino

briight commented 3 years ago

define SDDAT3 25 need to be added to the devkit part =)

/ Using the ESP32 DevKit with a screen module /

elif defined(ESP32DEVKIT)

define SCREEN_ROTATION 1

define FLIP_TOUCH_AXIS

define INVERSE_Y_TOUCH

define touchInterruptPin GPIO_NUM_27

define SDDAT3 25

else

error("Unsupported platform")

endif

sle118 commented 3 years ago

At this point, you might want to fork my repo and submit pull requests. Eventually, I think @DustinWatts would want to merge my code and make it official, so it is well spent efforts ;)

sle118 commented 3 years ago

Well I decided to finally take the plunge with a pull request. I have a couple more renames I'd like to do, but I am fairly certain that the current state of my code reached a state that is stable enough and yet offers very scalable user expansion capabilities with user actions. So there's the beginning of my contribution to this project : https://github.com/DustinWatts/FreeTouchDeck/pull/64

Once this gets merge, let's close this issue and continue the discussion on Discord

sle118 commented 3 years ago

Since the changes were merged, I'll go ahead and close this. Anyone interested, I'll be hanging around on Discord

DustinWatts commented 3 years ago

Amazing work!

briight commented 3 years ago

prob a good idea to wait for a easy way to config the menu's, been using urs for a while now and i love it thank you =)