search

meshtastic / firmware

Meshtastic device firmware
https://meshtastic.org
GNU General Public License v3.0
3.64k stars 914 forks source link

[Feature Request]: Updated Device Screen UI modes for all devices #2590

Open tropho23 opened 1 year ago

tropho23 commented 1 year ago

Platform

NRF52, ESP32

Description

Updated Device Screen UI modes

Overview of this proposal:

The current Meshtastic device screen user interface (UI) needs to be updated to display additional node and sensor information, enable enhanced user interaction without relying on BLE-paired mobile devices, and support advanced features introduced by new standalone devices integrating keyboards, rotary encoders, and matrix keypads. Device capabilities and new user interactions have grown beyond the capabilities of the current device screen UI.

Ideally, the current screen UI library should be able to support these proposed navigation flows and nested menus/actions for the new 128x64 screen UI modes; if not a new screen UI library must be considered. If currently supported e-ink displays can also take advantage of this proposal, that compatibility should also be accommodated if feasible.

For the Advanced Screen UI mode, a new screen UI/display library will need to be integrated to not only take advantage of TFT LCD and controllers, but also offer the graphical performance to support higher screen resolutions, interactive map displays, and possibly even touch interface, if feasible. The goal of the Advanced Screen UI mode is to minimize, or even eliminate the need to use BLE-paired mobile devices for any non-configuration use.

Selecting these modes will require a CLI/app/web configuration setting to choose Device Screen UI mode.

List of proposed Device Screen UI modes:

1. Boot Splash Screen: Shown during boot

  1. Add OEM/user configurable splash screen, either in place of Meshtastic default or show it after

2. Unset Region Screen: Shown when devices have region UNSET

  1. No changes recommended

3. Standard Screen UI: For devices without a user button or environmental sensors

  1. User can toggle displaying received messages on device screen:
    • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
    • If disabled: display default screen information
  2. Default screen shows local device information:
    • Device short name + last 4 characters of BLE MAC
    • Connected nodes (#/#)
    • GPS status (enabled/disabled, # of satellites)
    • Battery charge/percentage bar

4. Carousel Info Mode: For devices without a user button; continually rotates through device information

  1. User can toggle displaying received messages on device screen:
    • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
    • If disabled: display default screen information
  2. Default screen shows local device info:
    • Device short name + last 4 characters of BLE MAC (‘NODE 3f58’)
    • Connected nodes (#/#)
    • GPS status (enabled/disabled, # of sats)
    • Battery charge and percentage bar
  3. Additional info for the local device:
    • Environmental sensor data (if none, detect this and display "No sensors connected")
    • GPS-specific data (lat/long, dead reckoning direction arrow, # of satellites)
    • Battery state (charging/discharging) and voltage
    • Device role (client, router, relay, etc.) and firmware version
  4. Carousel mode continually cycles through items 1-3 above

5. Enhanced Screen UI: For devices with a user button

NOTE: Long press duration should be 2 seconds; 5+ second long press already initiates shutdown for nrf52 devices

  1. User can toggle displaying received messages on device screen:
    • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
    • If disabled: display default screen information
  2. Default screen shows local device info:
    • Device short name + last 4 characters of BLE MAC (‘NODE 3f58’)
    • Connected nodes (#/#)
    • GPS status (enabled/disabled, # of sats)
    • Battery charge and percentage bar
  3. Each single user button press cycles additional info for the local device:
    • Environmental sensor data (if none, detect this and display "No sensors connected")
    • GPS-specific data (lat/long, dead reckoning direction arrow, # of satellites)
    • Battery state (charging/discharging) and voltage
    • Device role (client, router, relay, etc.) and firmware version
    • 10 second timeout if no user input; returns to default screen (ideally configurable; 0=no timeout)
  4. Double user button press displays a list of connected mesh nodes:
    • Single user button press cycles through list of nodes
    • Double user button press selects desired remote node
      • Action 1: Send canned message
        • Press user button to select 'Send Canned Message'
        • Press user button again to cycle through list of canned messages
        • Double press user button to send canned message
        • <Once canned message is sent, automatically return to node list>
        • Long (2 sec) user button press * cancels/exits to node list
      • Action 2: Send waypoint
        • Press user button to select ‘Send Waypoint’
        • <Once waypoint is sent, automatically return to node list>
        • Long (2 sec) user button press cancels/exits to node list
      • Action 3: Request position update
        • Press user button to select 'Request position update'
        • <Once position update request is sent, automatically return to node list>
        • Long (2 sec) user button press cancels/exits to node list
      • Action 4: Display remote info
        • Press user button to select 'Display remote info'
        • Show default screen information described above (2a-2d) for selected remote node
        • Press user button again to display additional information described above (3a-3d) for selected remote node
        • Long (2 sec) user button press * cancels/exits to node list
      • Action 5: Admin actions
        • Press user button to select 'Admin actions'
        • Press user button again to cycle through list of admin actions (reboot, shutdown)
        • Double press user button to execute desired admin action
        • Long (2 sec) user button press exits to default local device screen at any time
        • 10 second timeout if no user input; returns to default screen
  5. Triple user button press disables GPS (if supported by device)

6. Standalone Screen UI: For devices with CardKB, rotary encoder, or matrix keypad

NOTE 1: For matrix keypads assign keys as desired for user button, CW, CCW, and preconfigured canned messages NOTE 2: Long press duration should be 2 seconds; 5+ second long press already initiates shutdown for nrf52 devices

  1. User can toggle displaying received messages on device screen:
    • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
    • If disabled: display default screen information
  2. Default screen shows local device info:
    • Device short name + last 4 characters of BLE MAC (‘NODE 3f58’)
    • Connected nodes (#/#)
    • GPS status (enabled/disabled, # of sats)
    • Battery charge and percentage bar
  3. Each single user button press cycles additional info for the local device:
    • Environmental sensor data (if none, detect this and display "No sensors connected")
    • GPS-specific data (lat/long, dead reckoning direction arrow, # of sats)
    • Battery state (charging/discharging) and voltage
    • Device role (client, router, relay, etc.) and firmware version
    • 10 second timeout if no user input; returns to default screen (ideally configurable; 0=no timeout)
  4. Double user button/rotary press (or CardKB Tab key):
    • Show 'Received Messages'
      • Press Enter or user button/rotary press to select 'Received Messages'
      • Subsequent rows show received messages, in descending chronological order (newest first)
        • Each row shows short name, date/time stamp, first few characters of message
        • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to choose the desired message
        • Press Enter or user button/rotary press to display selected message
        • Esc (CardKB) or long (2 sec) user button/rotary press to exit to Received Messages list
        • Esc (CardKB) or long (2 sec) user button/rotary press to exit to default screen
    • Show Node List
      • Navigating the node list:
        • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to cycle among remote nodes
        • Press Enter or user button/rotary press to select a remote node:
          • Action 1: Send a direct message (DM) (CardKB only)
            • Press Enter key to select 'Send DM'
            • Type message
            • Press Enter to send message
            • <Once DM is sent, automatically return to node list>
            • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
          • Action 2: Send a canned message
            • Press Enter key (CardKB) or user button/rotary press to select 'Send Canned Message'
            • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to choose the desired canned message
            • Press Enter or user button/rotary press to send canned message
            • <Once canned message is sent, automatically return to node list>
            • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
          • Action 3: Request position update
            • Press Enter key (CardKB) or user button/rotary press to select Request position update'
            • <Once position update request is sent, automatically return to node list>
            • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
          • Action 4: Display remote node info
            • Press Enter key (CardKB) or user button/rotary press to select 'Display Info'
            • Screen will show default screen information described above (2a-2d) for selected remote node
            • Enter key or user button/rotary press displays additional information described above (3a-3d) for selected remote node
            • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
          • Action 5: Admin actions
            • Press Enter key (CardKB) or user button/rotary press to select ‘Admin actions’
            • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to choose the desired admin action (reboot, shutdown)
            • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
          • To exit node list at any time:
            • Esc (CardKB) or long (2 sec) user button/rotary press
            • 10 second timeout if no user input; exits node list and returns to default screen
      • 10 second timeout if no user input; returns to default screen
  5. Quick method to send broadcast messages from main/default screen (CardKB only):
    • Type message
    • Press Enter to send message
  6. Quick method to send DMs from main/default screen (CardKB only):
    • Press Tab to show node list
    • Press Enter to select desired node
    • Type message
    • Press Enter to send DM
  7. Quick method to send canned messages to a selected node from main/default screen (CardKB only):
    • Press Tab to show node list
    • Press Up/Down arrow key to highlight desired node
    • Press Enter to select desired node
    • Press Up/Down arrow key to highlight desired canned message
    • Press Enter to send canned message
  8. Quick method to send canned messages to Broadcast (all nodes) from main/default screen (CardKB only):
    • Press Up/Down arrow key to highlight desired canned message
    • Press Enter to send canned message
  9. Triple user button press disables GPS (if supported by device)

7. Advanced Screen UI: For devices with TFT LCD screen (TBD)

8. Basic Device Settings: For Standalone Screen UI and Advanced Screen UI only; show essential settings user can change without requiring a paired phone

  1. Name: set/change long name and short name
  2. BLE: enable/disable, set/change pairing PIN
  3. Notifications: enable/disable external notifications (LED, buzzer, vibramotor; each independently toggled)
  4. GPS: enable/disable (same as triple user button press), toggle GPS smart position updates, change GPS update interval
mverch67 commented 1 year ago

Great proposal!

Another nice option for the ultimate meshtastic device experience:

  1. Enhanced Device Settings: For Advanced UI with touch screen only; full device configuration without ever requiring a paired phone or web/cli.

BTW: I started working on (7) driver configuration (lovyangGFX library), I have the following color TFT screens available for configuration and tests:

tropho23 commented 1 year ago

Great suggestion! I will add that to my list of proposed device screen UI modes, it's great.

Also thank you for the LCD/TFT development, it's super exciting to imagine what we can do with that.

GUVWAF commented 1 year ago

I would also like to propose having a way to show whether a sent message was acknowledged. I don't think it's worth to store transmitted messages as well; a small pop-up when the last message was acknowledged should be enough.

geeksville commented 1 year ago

This issue has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/canned-messaging-see-list-of-recent-messages/8060/2

Nikguy321 commented 1 year ago

Yeah I think being able to see a small list of like the last 30-40 text messages sent (instead of just the most recent) would be really nice. Would make it a lot more viable to use canned messaging instead of having to have a phone attached.

YAMLcase commented 1 year ago

I'm glad this issue was proposed. I've been tinkering with the T-deck and thought about how we might replicate the android meshtastic client onto the device

geeksville commented 1 year ago

This issue has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/t-deck-from-lilygo/7679/74

TheCommsChannel commented 1 year ago

I would love to see this as I have a T-Deck as well.

ko4whd commented 1 year ago

Best part about the T-Deck is the TF Card slot. I know this could play a huge role in future updates and options either for additional storage or a full custom OS (ex... ESP32Berry or Raspberry Pi OS Lite ...etc)

ko4whd commented 1 year ago

@tropho23 There is a guy on Youtube (@ThatProject) stated he would be able to help the Meshtastic Team in the development of the Meshtastic app on the T-Deck with FreeRTOS if any team members could reach out to him and let him know what he needs to do to assist.

Nikguy321 commented 1 year ago

There is a guy on Youtube (@ThatProject) stated he would be able to help the Meshtastic Team in the development of the Meshtastic app on the T-Deck with FreeRTOS if any team members could reach out to him and let him know what he needs to do to assist.

Is he the guy that does espberry? That's a nice software.

ko4whd commented 1 year ago

@Nikguy321 that's the guy

danumbaman commented 11 months ago

I would love to see a way to load openstreet maps saved to Sd-card on T-Deck.

thatSFguy commented 8 months ago

This is a very well written epic, since it is way larger than a feature. I'm not sure how the meshtastic developers decide what they work on, however, I think it would be more likely taken up if this is split up into individual features. I'm mostly interested in #6 at this point, but I know having the complete vision helps a developer to avoid rework. Again, a very well written request. 👏

KyleMaas commented 7 months ago

I'd love to be able to help with this effort, but not sure where to start. I keep seeing references to the new UI branch, which doesn't look like it's active, and references to 3.0, but again can't find anything current on that. I don't want to go about taking on such a massive architectural change without substantial guidance, and ideally not duplicating someone else's efforts. So what's the best way for me to start helping with this?

Murturtle commented 6 months ago

My idea is similar to yours but more friendly to single button devices.

https://docs.google.com/presentation/d/16ZLfRwzQSzDY1iqpp2x5R5QlF5D-X9AmSsojUy412MU/edit?usp=sharing

mverch67 commented 6 months ago

My idea is similar to yours but more friendly to single button devices.

https://docs.google.com/presentation/d/16ZLfRwzQSzDY1iqpp2x5R5QlF5D-X9AmSsojUy412MU/edit?usp=sharing

The idea for navigation is good, but I would really try to avoid the double-click because its support introduces massive lagging of the UI navigation and also false clicks. I think for that reason lvgl also does not support double-click. Instead, simply use a "back" entry in the list.

rcarteraz commented 6 months ago

My idea is similar to yours but more friendly to single button devices.

https://docs.google.com/presentation/d/16ZLfRwzQSzDY1iqpp2x5R5QlF5D-X9AmSsojUy412MU/edit?usp=sharing

I like the idea of a new multi-screen UI for OLED devices, but I personally I think some of what you are proposing is a UX nightmare and extremely cluttered.

Murturtle commented 6 months ago

My idea is similar to yours but more friendly to single button devices. https://docs.google.com/presentation/d/16ZLfRwzQSzDY1iqpp2x5R5QlF5D-X9AmSsojUy412MU/edit?usp=sharing

The idea for navigation is good, but I would really try to avoid the double-click because its support introduces massive lagging of the UI navigation and also false clicks. I think for that reason lvgl also does not support double-click. Instead, simply use a "back" entry in the list.

Makes sense now that I think about it.

todd-herbert commented 6 months ago

Just to make sure it doesn't get missed, I'm hoping to start a discussion about which direction E-Ink displays should go in the future: https://github.com/meshtastic/firmware/discussions/3908

kalestew commented 5 months ago

Really enjoy this discussion and the progression of the meshtastic UX. Thanks to everyone contributing.

I think one thing that is missing in prominent display in the firmware it deserves is channel utilization. For a single screen status view, I would consider this almost as important as the battery voltage.

jeffhuys commented 6 days ago

With all the scares about blackouts and the like, and my phone service and home internet becoming more unreliable, me and a lot of people around me are looking for ways to get off the grid. This brought me to the T-Deck and eventually here. I want to be able to communicate with my family in emergencies.

I would like to stress that having a good UX, or even just a few small improvements, could be a catalyst for meshtastic and this whole industry. I sincerely hope that this is seen for what it is; it could be a very important crossroad. Please, don't miss it!

I will keep using the T-Deck and try to help with improving the meshtastic firmware as I do have a background in embedded systems.