Flexible configuration

[sle118]

This is just a comment. @DustinWatts and the other contributors to this project. Feel free to close it.

I wanted to thank you for taking the time to build this. It was exactly what I was looking for!

I took the time to port the code to the TTGO T-Watch 2020 v1 which I happen to own. This wasn't a huge effort; the watch has the same capacitive touch controller that the project uses. A couple of notable differences are the screen resolution (smaller) as well as the onboard power management chip which needs to be told to power on in order to get the back light to work.

Just for fun, I spent a couple of hours refactoring a bit of the code, especially around how menus/screens are represented and loaded internally. I've done it in a way that would allow any number of menus to be created, each with an arbitrary number of buttons, and an arbitrary number of actions, all the while preserving the JSON format which you are currently using (for the sake of staying compatible with existing installs).

Essentially, there are now a few more classes:

• Action: defines an individual action with its type (special keys, media keys,letters, etc) and associated values (int or char)
• Button: defines a given button with its type (latch, standard, menu), as well as its latch status, both logos, etc. They are responsible to determine if they need to draw their content or not (based on status change, for example). Menu buttons are used to activate a different menu. For example button 1 on the home screen activates menu1. Eventually, menu buttons should be allowed in a sub menu (e.g. menu1 ->menu11, menu12 , etc. )
• Menu: defines a group of buttons. Menus will trigger buttons to draw themselves, calculating the proper position and size on screen. The positions will be dynamically calculated at load time, based on the screen resolution and the number of buttons on a given menu.

Let me know your thoughts and if this is something which you'd like to see some day.

[DustinWatts]

@sle118 Sorry for the late response. I was very busy with selling ESP32 TouchDowns this week ;)

But this is amazing! Number 1 on my priorities list was refactoring the code in a way that it would allow for more actions. I know this is a much requested feature. So in answer to your question, i definitely like to see that! There is a branch called development. Which my idea was to try out all these new things. A branch where I don't have to be afraid that a pull request may break things. I invite you to share your code there!

[sle118]

ok. Noted. I did quite a bit of refactoring to the code so it might take you a bit of time to adjust to the new reality ;)

I'm still in the very early stage and I just reached a clean compile with a ton of "//todo: " in the code to implement the features I had to butcher in order to start doing some surface testing.

It took me some time to adjust to the Arduino environment; I am used to the more bare bone environment provided by the ESP-IDF.

[DustinWatts]

If you have anything to share let me know! Looking forward to it :)

[sle118]

I will start by reaching a minimum level of stability (e.g no crash/boot loop) and some basic navigation before sharing. But here's a glimpse of the FTAction class and how it leverages arrays to reduce code redundancy.

Definitions:

using namespace std;
#define MEDIA_2_VECTOR(m)  {m[0],m[1]}
namespace FreeTouchDeck {

    static const uint8_t ArrowsAndTab[]={KEY_UP_ARROW,KEY_DOWN_ARROW,KEY_LEFT_ARROW,KEY_RIGHT_ARROW,KEY_BACKSPACE,KEY_TAB,KEY_RETURN,KEY_PAGE_UP,KEY_PAGE_DOWN,KEY_DELETE} ;
    static const vector<vector<uint8_t>> Keys  = {  {
        MEDIA_2_VECTOR(KEY_MEDIA_NEXT_TRACK),
        MEDIA_2_VECTOR(KEY_MEDIA_PREVIOUS_TRACK),
        MEDIA_2_VECTOR(KEY_MEDIA_STOP),
        MEDIA_2_VECTOR(KEY_MEDIA_PLAY_PAUSE),
        MEDIA_2_VECTOR(KEY_MEDIA_MUTE),
        MEDIA_2_VECTOR(KEY_MEDIA_VOLUME_UP),
        MEDIA_2_VECTOR(KEY_MEDIA_VOLUME_DOWN),
        MEDIA_2_VECTOR(KEY_MEDIA_WWW_HOME),
        MEDIA_2_VECTOR(KEY_MEDIA_LOCAL_MACHINE_BROWSER),
        MEDIA_2_VECTOR(KEY_MEDIA_CALCULATOR),
        MEDIA_2_VECTOR(KEY_MEDIA_WWW_BOOKMARKS),
        MEDIA_2_VECTOR(KEY_MEDIA_WWW_SEARCH),
        MEDIA_2_VECTOR(KEY_MEDIA_WWW_STOP),
        MEDIA_2_VECTOR(KEY_MEDIA_WWW_BACK),
        MEDIA_2_VECTOR(KEY_MEDIA_CONSUMER_CONTROL_CONFIGURATION),
        MEDIA_2_VECTOR(KEY_MEDIA_EMAIL_READER)
    }};

    static const uint8_t OptionKeys[]={
        KEY_LEFT_CTRL,
        KEY_LEFT_SHIFT,
        KEY_LEFT_ALT,
        KEY_LEFT_GUI,
        KEY_RIGHT_CTRL,
        KEY_RIGHT_SHIFT,
        KEY_RIGHT_ALT,
        KEY_RIGHT_GUI        
    };
    static uint8_t FunctionKeys[] = {
        KEY_F1,
        KEY_F2,
        KEY_F3,
        KEY_F4,
        KEY_F5,
        KEY_F6,
        KEY_F7,
        KEY_F8,
        KEY_F9,
        KEY_F10,
        KEY_F11,
        KEY_F12,
        KEY_F13,
        KEY_F14,
        KEY_F15,
        KEY_F16,
        KEY_F17,
        KEY_F18,
        KEY_F19,
        KEY_F20,
        KEY_F21,
        KEY_F22,
        KEY_F23,
        KEY_F24                
        };
    static uint8_t Helpers[] = {
        KEY_F1,
        KEY_F2,
        KEY_F3,
        KEY_F4,
        KEY_F5,
        KEY_F6,
        KEY_F7,
        KEY_F8,
        KEY_F9,
        KEY_F10,
        KEY_F11    
        };        
    static const vector<vector<uint8_t>> Combos = {{
        {KEY_LEFT_CTRL,KEY_LEFT_SHIFT},
        {KEY_LEFT_ALT,KEY_LEFT_SHIFT},
        {KEY_LEFT_GUI,KEY_LEFT_SHIFT},
        {KEY_LEFT_CTRL,KEY_LEFT_GUI},
        {KEY_LEFT_ALT,KEY_LEFT_GUI},
        {KEY_LEFT_CTRL,KEY_LEFT_ALT},
        {KEY_LEFT_CTRL,KEY_LEFT_ALT,KEY_LEFT_GUI},
        {KEY_RIGHT_CTRL,KEY_RIGHT_SHIFT},
        {KEY_RIGHT_ALT,KEY_RIGHT_SHIFT},
        {KEY_RIGHT_GUI,KEY_RIGHT_SHIFT},
        {KEY_RIGHT_CTRL,KEY_RIGHT_GUI},
        {KEY_RIGHT_ALT,KEY_RIGHT_GUI},
        {KEY_RIGHT_CTRL,KEY_RIGHT_ALT},
        {KEY_RIGHT_CTRL,KEY_RIGHT_ALT,KEY_RIGHT_GUI}
    }};

and the embryo of the execute method

  void FTAction::Execute()
    {
        Serial.println("[INFO]: BLE Keyboard action received");
        switch (Type)
        {
            case ActionTypes::NONE:
            break;

            case ActionTypes::DELAY:
                delay(value);
            break;
            case ActionTypes::ARROWS_AND_TAB:
                bleKeyboard.write(ArrowsAndTab[value-1]);
            break;
            case ActionTypes::MEDIAKEY:
                for(auto kp : Keys[value-1])
                {
                    bleKeyboard.write(kp);
                }

            break;
            case ActionTypes::LETTERS:
            case ActionTypes::SPECIAL_CHARS:
                bleKeyboard.print(symbol);
            break;
            case ActionTypes::OPTIONKEYS:
                if(value-1 <sizeof(OptionKeys)/sizeof(uint8_t))
                {
                    bleKeyboard.press(OptionKeys[value-1]);
                }
                else 
                {
                    bleKeyboard.releaseAll();
                }
            break;
            case ActionTypes::FUNCTIONKEYS:
                bleKeyboard.press(FunctionKeys[value-1]);
            break;
            case ActionTypes::NUMBERS:
                bleKeyboard.print(value);
            break;

            case ActionTypes::COMBOS:
                for(auto k  : Combos[value])
                {
                    bleKeyboard.press(k);
                }
            break;
            case ActionTypes::HELPERS:
                if(generalconfig.modifier1 != 0){
                    bleKeyboard.press(generalconfig.modifier1);
                }
                if(generalconfig.modifier2 != 0){
                    bleKeyboard.press(generalconfig.modifier2);
                }
                if(generalconfig.modifier3 != 0){
                    bleKeyboard.press(generalconfig.modifier3);
                }
                bleKeyboard.press(Helpers[value-1]);
                bleKeyboard.releaseAll();
            break;
            case ActionTypes::MENU:
            break;
            case ActionTypes::LOCAL:
                switch ((LocalActionTypes)value)
                {
                case LocalActionTypes::ENTER_CONFIG:
                    /* code */
                    break;
                case LocalActionTypes::BRIGHTNESS_DOWN:
                    if (ledBrightness > 25)
                    {
                        ledBrightness = ledBrightness - 25;
                        ledcWrite(0, ledBrightness);
                    }                
                break;
                case LocalActionTypes::BRIGHTNESS_UP:
                    if (ledBrightness < 230)
                    {
                        ledBrightness = ledBrightness + 25;
                        ledcWrite(0, ledBrightness);
                    }
                break;
                case LocalActionTypes::SLEEP:
                    if (generalconfig.sleepenable)
                    {
                        generalconfig.sleepenable = false;
                        Serial.println("[INFO]: Sleep disabled.");
                    }
                    else
                    {
                        generalconfig.sleepenable = true;
                        Interval = generalconfig.sleeptimer * 60000;
                        Serial.println("[INFO]: Sleep enabled.");
                        Serial.print("[INFO]: Timer set to: ");
                        Serial.println(generalconfig.sleeptimer);
                    }                
                break;
                default:
                    break;
                }
            break;

        default:
            break;
        }

    }

I have implemented a std::queue which receives actions. For example, pressing a button will result in a new action being inserted in the queue. Then the queue is processed in the loop()

  while (!Queue.empty())
  {

    ResetSleep();
    Queue.front().Execute();
    Queue.pop();
  }

[DustinWatts]

So far that is looking great!

[sle118]

I'm thinking of implementing something similar to AutoHotkey (a reduced set to begin with) to handle sending keypresses: https://www.autohotkey.com/docs/commands/Send.htm

This would be a complementary method to the ones that are currently implemented so backwards compatibility is preserved.

[coryb]

I missed this issue, seems like we are working on similar problems. I just created PR #52 that makes the menus sizing configurable, not sure how incompatible this change will be with the work you are doing. I didnt touch much of the action logic, so maybe we can use the work you have done there? Or if your code is in a good state we can cancel my PR and go with your work?

[sle118]

Well, this is annoying; it looks like the Arduino flavor of the esp-idf is stuck in the past and it's missing several of the newer c++ std features. For instance, there's no std::unique_ptr to reduce the memory footprint of vectors, lists, queues, etc. I'll see how I can work around that, given that Arduino has a broader reach with tinkerers.

[DustinWatts]

I missed this issue, seems like we are working on similar problems. I just created PR #52 that makes the menus sizing configurable, not sure how incompatible this change will be with the work you are doing. I didnt touch much of the action logic, so maybe we can use the work you have done there? Or if your code is in a good state we can cancel my PR and go with your work?

Maybe we can keep that PR open and integrate it to what @sle118 is working on?

@sle118 That is annoying, off course for a pre-build binary this wouldn't matter. But I like to keep the whole source accessible for everyone to tinker with as you stated.

I Appreciate both your work!

[sle118]

It is a rather complete refactoring of the effort and unlikely to be compatible with existing pull requests. I will most likely push the changes to my own repo (I forked this repo) so everyone can have a look.

[coryb]

I will try out your fork when you get it pushed to see how it compares to mine, having just gone through all this I am at least pretty familiar with the codebase now :) So maybe I can help out.

[sle118]

I'm using a TTGO T-Watch for testing and I have basic navigation, pressing/releasing, and latch working albeit still with some glitches with transparency, which I am just starting to analyse in the original code base. Next I'm going to implement the configuration menu, for which actions are already implemented.

I have ordered the following, which is going to be easier to debug using JTAG than the watch: http://www.makerfabs.com/esp32-3.5-inch-tft-touch-capacitive-with-camera.html

I don't need the camera, but it was cheap enough to try out.

Once I am satisfied with the results, I'll push the code to my repo. @coryb I don't plan on fine tuning all the display code, but it will be in a sufficiently good enough state for you to step in if you'd like.

Down the road, I am seriously thinking of merging the core functionality with the platform I created for a Logitech media (aka squeezebox) music streamer which I developed with friends. You can see for yourself here: https://github.com/sle118/squeezelite-esp32

This provides a number of features:

• full OTA capabilities from releases created on GitHub with actions or through web upload
• Wi-Fi manager which sets up an access point to allow initial configuration of the network
• key value pair configuration management that is thread safe
• ability to fully use external memory; using psram from the wrover (if equipped) would dramatically improve screen refresh time
• flexible UI which can be used to configure various system options like gpio's, audio output type (i2s or PWM chiptune), etc, making it possible for a single release binary to target various hardware platforms -etc

As for the UI engine, littlevgl could be an interesting component to consider.

Of course this would be a departure from the current full Arduino based approach, with bare but unleashed esp-idf. However, the Arduino platform can still be used (as a component), which means existing Arduino libraries would still work.

Anyhow, I'm just dumping some high level ideas here. I don't expect that this project would follow through such a radical change, but nevertheless I'll always keep the code available for anyone to play with.

[danielhunt]

As it happens I have also been working on making the menu system more flexible having just received my device in the post yesterday.

My own version is very similar to what @coryb has come up with - but as I am not a true C++ dev it's far less neat.

I might be interested in helping out on the frontend side of things once you have all settled on a valid codebase to run with. Right now, #52 looks like it will be a good starting point but if @sle118 has completely overhauled the codebase I'm not sure there's much point in moving forward on anything yet.

[sle118]

This is getting real funny!

Since I do not own the hardware, I will leave it to the group to decide which way they want to go. I have limited bandwidth and definitely cannot resolve all the issues that I have introduced with my changes!

There are many areas that I wanted to tackle but didn't get to.

[sle118]

allright. I made good progress tonight and had navigation up and running. I have pushed whatever I have so far here https://github.com/sle118/FreeTouchDeck

Note: I'm using VSCode with the Arduino extension, rather than the "pure" Arduino IDE. Also Note: This WILL crash on you; after getting navigation right, I decided to start making some performance improvements. The first move was to cache logos (bitmaps) in PSRAM and render from PSRAM. Unfortunately, this was a bold change and I haven't yet come to resolving the crash. I'm pushing the changes so @danielhunt @coryb and @DustinWatts can have a look and see if you like what you see.

note that this is only addresses PART of the backend and there's still quite a bit of change to be made to de-couple the code from the hardware but it's a good start (as far as I can tell).

Please have a look and comment.

thank you!

[danielhunt]

Please have a look and comment.

The diff between your code and master is so far beyond my abilities to understand 😢 :D

[DustinWatts]

Please have a look and comment.

thank you!

No thank you! :) This is a massive change and it looks great so far! I have to take some time going through it but this weekend I hope to have some time and will try it out (accepting crashes ;) ). But I like where you are going.

[sle118]

right now it does crash on start because of my last minute change, leveraging PSRAM if present. I'll try to remediate so you have a somewhat better experience. I am not sure if it will compile on the Arduino IDE, but I think it should, given that VSCode calls it in the backend when building.... it's super super slow compared to native ESP-IDF v4.0, which leverages cMake + Ninja to speed up the build.

[sle118]

Please have a look and comment.

The diff between your code and master is so far beyond my abilities to understand 😢 :D

Don't read with the old code in mind; read it as if you were trying to understand another piece of code.

[sle118]

One more note here. I think I have found out the door cause of the crashes. I will have to overwrite the new operator to allocate memory in psram.

Can anyone confirm that the original hardware has a wrover or psram chip on board?

Edit: well well... I just found that the touch down hardware doesn't have the luxury of psram. So I'm going to have to rewrite portions of the code to setup arrays in the code space instead of dynamically allocated structures as I'm doing right now.

I should also receive my new test board, which will allow me to debug with JTAG instead of fishing for bugs using serial output.

Thank you for your patience!

[sle118]

Update: I just pushed some commits which restore basic navigation. It is no longer crashing! Note that I lost border drawing around buttons and latch mode too. This shouldn't be a major thing to resolve. It turns out using std::vector wasn't a good idea since it stores all data in a contiguous memory location. This means every time I was pushing a new element, the system had to reallocate a new block of memory, possibly resulting in memory fragmentation.

[DustinWatts]

This is going amazing @sle118 ! Can you DM me your contact details? dustin_watts@yahoo.com I can arrange a ESP32 TouchDown for you so you have the exact hardware.

[danielhunt]

@sle118 while building this I get the following error:

FreeTouchDeck:360:43: error: 'CONFIG_ARDUINO_LOOP_STACK_SIZE' was not declared in this scope
   Serial.printf("Main Task size is %d\r", CONFIG_ARDUINO_LOOP_STACK_SIZE);

I presume I need to have another library or something for this to work?

[sle118]

@danielhunt I will push some more changes today. Some minor changes were done, but now a "back button" action is possible. Useful for multi level menus!

Edit: changes pushed. Let me know if they compile now.

[DustinWatts]

@sle118

Got a few warnings (mostly unused variable, so no big deal). But did get an error. Line 284 of FTAction.cpp:

if (SetActiveScreen)
            {
                ESP_LOGD(module, "Selecting menu %s\n", symbol);
                SetActiveScreen(symbol);
            }

will always evaluate as true. So, just removing the if statement and it compiles. Next step is to actually upload to see it in action :)

[sle118]

Which Arduino environment are you "officially using"? I'll make sure I compile it there first before committing anything. I've been using vscode with the Arduino extension because it is far better than the basic Arduino IDE and because I'm used to it, but I read that the beta Arduino environment is an improvement over the old one

[DustinWatts]

Offially I use the basic Arduino IDE 1.8.13. But I regularly switch between that and VS code myself too. I make sure it compiles in the basic Arduino IDE because that is what most users use. I also so set compiler warnings to "All". Just in case someone has this set. It then halts on things like not returning anything in a boolean function.

[sle118]

Later today, I'll add a formal back button icon (from the material design, white on black 75x75). The menu navigation now keeps a reference to the previous menu when navigating down menus. All screens/menus with user buttons will now have this back button instead. For screens without buttons (e.g. the information display), a screen level action of "back" is added. This means when user presses such a button less screen, it is sent back to the previous screen that was being displayed.

The compile problem should also be resolved; to help with cross platform (if you ever come up with a new design that has... an i2s audio chip for example), some action implementation callbacks are now part of a structure. So you have a callback structure in FTAction.h

    typedef bool (*ActionCallbackFn_t)(FTAction *);
    typedef struct ActionCallbacks
    {
        ActionCallbackFn_t RunLatchAction;
        ActionCallbackFn_t ChangeBrightness;
        ActionCallbackFn_t PrintInfo;
        ActionCallbackFn_t ConfigMode;
        ActionCallbackFn_t SetActiveScreen;

    } ActionCallbacks_t;

    ActionCallbacks_t *ActionsCallbacks();

and the structure is initialized in the main sketch

  ActionsCallbacks()->PrintInfo=printinfo;
  ActionsCallbacks()->ChangeBrightness=ChangeBrightness;
  ActionsCallbacks()->ConfigMode=ConfigMode;
  ActionsCallbacks()->RunLatchAction = RunLatchAction;
  ActionsCallbacks()->SetActiveScreen=RunActiveScreenAction;

a handy call wrapper is defined globally. It checks if the pointer was assigned to a function, and if so calls it. If it's not implemented, a warning is written to serial. #define EXECUTE_IF_EXISTS(x,y) if(x) { x(y); } else {ESP_LOGW(module,"Function %s not implemented", QUOTE(x));}

Some examples of how the FTAction class invokes these

       case ActionTypes::MENU:
            EXECUTE_IF_EXISTS(callbacks.SetActiveScreen,this);
            break;
        case ActionTypes::SETLATCH:
        case ActionTypes::CLEARLATCH:
        case ActionTypes::TOGGLELATCH:
            EXECUTE_IF_EXISTS(callbacks.RunLatchAction,this);
            break;

[danielhunt]

FYI I know you're still actively working on this but I get stuck in a bootloop when I upload this latest version.

During boot I see:

- Screen flashes white, then black
- "Loading version: 0.9.11" is display in the top-left corner
- Then a line of small boxes across the top of the screen (5.5 boxes are shown - the 6th box is cut off at the side of the screen)

[sle118]

@danielhunt For now, you can try to change the core debug level (logging) to debug. I stumbled on the same issue when I compiled with logs at level info and found the root cause. It'll be fixed on my next push.

[sle118]

I pushed the WIP for today. Note that 2 new images were added (back button, etc). Please flash the data partition before trying. Some changes:

• Latching is now "semi working", but needs love.
• Back button replaces home button. This opens the door to multi level/context menus where you could have, say, a button on the home menu to get to a steaming menu where each button can either go to another menu, or execute a task
• Consolidated action callback registration. This should prevent compiler errors and clarify what needs to be implemented for the actions to work. Of course, all callbacks are already hooked up
• fix issue where compiling with logging level info caused icons to disappear
• new functionality: it is possible to define a button that has no logo. Doing so will have the system draw an empty bitmap with the button label text printed on screen.
• better error handling for configuration mode. This should simplify troubleshooting if I broke anything in that area
• reduced the screen action thread stack size

Now, and to get to the point of this discussion, I think a different configuration schema should replace the existing one. Since my changes already support reading it as it is, I could actually implement an automated mechanism to convert to the new one. I was thinking of something like this

[
   {
      "homescreen":[
         {
            "buttonname":"audio",
            "type":"MENU",
            "logo":"music.bmp",
            "actions":[
               {
                  "type":"MENU",
                  "value":"audio"
               }
            ]
         },
         {
            "buttonname":"obs",
            "type":"MENU",
            "logo":"obs.bmp",
            "actions":[
               {
                  "type":"MENU",
                  "value":"obs"
               }
            ]
         }
      ]
   },
   {
      "audio":[
         {
            "buttonname":"mute",
            "type":"LATCH",
            "logo":"mute.bmp",
            "latchedlogo":"muted.bmp",
            "actions":[
               {
                  "type":"MEDIAKEY",
                  "value":"1"
               }
            ]
         },
         {
            "buttonname":"volumedown",
            "type":"STANDARD",
            "logo":"volumedown.bmp",
            "actions":[
               {
                  "type":"MEDIAKEY",
                  "value":"2"
               }
            ]
         },
         {
            "buttonname":"volumeup",
            "type":"STANDARD",
            "logo":"volumeup.bmp",
            "actions":[
               {
                  "type":"MEDIAKEY",
                  "value":"3"
               }
            ]
         }
      ]
   }
]

[DustinWatts]

Some great WIP Sébastien! If I do not set Werror, things compile and run without crashes. For me capacitive touch (the FT6236) doesn't work yet. I haven't really looked in to it yet.

A few things I noticed:

• BLE Keybaord actions are not being sent
• After going to the INFO page and back, the back button draws a blank screen
• Latching on works, but latching off doesn't

But again, great work!

[sle118]

I haven't reached testing actual actions, since I've been refactoring menu navigation, etc. If actual actions are sent, then it is mostly by luck; I've implemented the logic but not tested it!

I'm surprised that non capacitive code works, since I don't have a test system and since I've mostly been working around it trying to preserve what was there. I'd be curious to see if you get the calibration prompt on initial start? Which board are you selecting when you build? I will try to compile and see how the defines deactivate code. Do you see anything unusual in the serial output? I'm currently testing on that TTGO T-Watch, which also happens to have the ft6236 chip, but it reports a different chip id and I had to slightly modify the library to accommodate

[DustinWatts]

Nothing weird on the serial monitor.

The restive touch is a function of the button handling class use by TFT_eSPI, so if you draw a button, a touch area is automatically created. And I do get the calibration screen (TouchCalibrate() is the function). The capacitive touch doesn't need calibration so that is skipped if #USECAPTOUCH is defined.

The board I'm building with is a regular ESP32 Dev Kit. A WROOM-32D to be precise. The capacitive touch chip I use is the FT6236 (also FT6202 only different chip ID) and the library I use is mine: https://github.com/DustinWatts/FT6236. The ESP32-BLE-Keyboard I'm using is a for of the library by T-vK: https://github.com/DustinWatts/ESP32-BLE-Keyboard. Which I made some changes to, mainly an .end() function, decreasing the memory usage, and added some keys.

But don't worry too much, it is not difficult to match a touch to the boundaries of a button. So I'll have a dig around which will also help me to dive in the code a bit more ;)

[sle118]

Ah, I think it might have to do with rotation. I might have removed your map logic by mistake. I'll see if that is the case. The T-Watch has a 1:1 mapping between the screen and the capacitive touch so I tweaked the code a bit in that area,

[danielhunt]

I'm looking to try using TJpg_Decoder to use JPG files instead of BMP ones (the /data/logo directory goes from 952k to 108k without looking at additional compression. This makes a material difference to the data directory size 😱 )

@sle118 your abstraction of the BMP classes is 👌 but bmp as a concept is still littered through the codebase (default images and like) Would it be worth abstracting this away a bit more so that a JPGImage class could be easily swapped into place instead of BMPImage?

[sle118]

@danielhunt we might have stack/memory issue decoding JPG unless you have a very efficient decoder ;)

[danielhunt]

This is my first ESP32 experience so I know nothing. At all.

But I do know that I need to upload 1.2 megs of data to a 2 meg partition on a 4 meg memory chip ... which isn't great :)

[sle118]

@danielhunt the biggest concern (AFAIK) would be to find a decoder that can run in a relatively small stack. There's a good chance it will work, but memory is a heck of a constraint here. I am used to the "luxury" of 4Mb PSRAM that we get with a chip like the Wrover, but the hardware platform that this project support only has 500Kb or so of RAM and it is very limiting.

All in all, I think the library you identified would work. However, there's still much to do with my refactoring effort before calling it victory. I've hit a memory wall and I am considering a reduction in a few areas where data is static and can be pulled from flash. Think of this like "swapping" to disk, except this would be swap once, read many.

[sle118]

@DustinWatts I have hit a memory wall right now and I'm checking to see if there is anything I can do to optimize the code; attempting to start configuration in AP mode fails, likely due to running out of IRAM. When everything is finished loading, there's only 85kb left or so and attempting to load the AP fails miserably. Is this something you ever encountered?

[sle118]

I just pushed another change, by the way

[danielhunt]

I switched to verbose logging and my monitor output was flooded with content like this (in an infinite loop). Is this expected? Could it explain your memory issues?

[sle118]

@danielhunt this is most likely the action queue thread, which is running in parallel and never ends, so it is expected. When you compile at a certain level, all of entries below that are discarded by the compiler, and verbose is definitely not recommended when actually running the thing.

As for the memory issue, it isn't a leak or something like this; it is just that the memory is low after loading everything. I already made some changes that improve things by a good margin, but I haven't solved the configuration crash yet

[sle118]

@danielhunt would you be in for a frontend/backend collaboration to handle the "new generation" configuration? Thinking that we might be able to leverage google's blocky for this? https://developers.google.com/blockly

Or not... I've been wondering how to make a flexible interface as simple enough for end users and I am definitely not a front end/ui person!

[sle118]

Would it be worth abstracting this away a bit more so that a JPGImage class could be easily swapped into place instead of BMPImage?

I renamed the class ImageWrapper :)

[sle118]

I am still having an issue when starting the configuration mode with default access point; the asynchronous web server is unable to start and there's nothing simple I can do around that; many of these packages insist on having their own task and there isn't much control in the Arduino to find tune memory usage of the underlying esp-idf.

Note that this could be a side effect of my platform, the TTGO T-Watch, which has psram. Chances are that the platform is configured differently with the result of less memory being available right from start.

So I'm starting to think that my best bet would be to position a simple indicator in the RTC memory area of the esp32. Starting the configuration mode would look like:

• set the indicator to "config mode"
• reboot the esp32
• in the first lines of the setup method, look at the reboot reason and if soft reset check the indicator
• load general config and Wi-Fi config, initialize hardware
• then if the indicator is set to config mode, reset it, jump straight to config and skip all the rest of the menu loading
• if the indicator is not set, then boot as usual

[DustinWatts]

I am still having an issue when starting the configuration mode with default access point;

This is something I ran in to a lot when starting development. The libraries used, use a lot of RAM. I finally got it to a stage where it runs stable, but a lot of compromises were made. I think some kind of "boot flag" is a good idea if there is no way around it. I hope when you get your ESP32 TouchDown (which is the same WROOM as on a DevKit) you can get some different results!

Btw, deleting the BLE task and stopping BLE completely did the tick for me when I ran in to the webserver issue...

[sle118]

With the dedicated reboot mode, the system is now rock solid and there's no need to worry about stopping extra services. The system mode flag is sitting in a memory area that doesn't get initialized on reboot. __NOINIT_ATTR

I also just updated the button background colors. So menu colors and function buttons colors now show up correctly!

The only thing I haven't yet gotten to work is the mute/volume control (media buttons). Next for me will be some code cleanup and some minor refactoring. Then I'm thinking of implementing "toJSON()" methods that are callable from the "menu" level, as well as new json constructors the new json format.

[sle118]

@DustinWatts Have you considered the possibility of a multi-stage power save mode?

• After shorter period of time, enter light sleep and power down or reduce to a minimum the PWM of the LCD back led
• enter light sleep
• After the existing long period, enter deep sleep

[sle118]

Ah, and I also ditched my 2 action worker threads. There just isn't enough to do to justify the added overhead!