search

meshtastic / firmware

Meshtastic device firmware
https://meshtastic.org
GNU General Public License v3.0
3k stars 718 forks source link

[Feature Request]: Last X messages storage and sync with APPs/devices #3664

Closed Dorn8010 closed 2 months ago

Dorn8010 commented 2 months ago

Platform

ESP32

Description

The great network has the biggest problem on the "last mile". The messages arrive at my devices (ESP32 and NRF52) and are display on the LORA devices. But although the e.g. last message is display, there is no reliable sync with the APP / Webinterfaces. So if I'm not constantly connected to my LORA device, most messages get lost.

My proposal :

This assures that a consistent chat is available on all devices even if they are not connected continuously and the storage is managed.

Since I think this would massively improve the reliability and ease of use of MeshTastic I would be happy to see this option soon. If I can support with Phyton coding or testing I would be happy to help.

Kind regards and keep the great work going !!!!

thebentern commented 2 months ago

We actually already have a queue structure that holds unreceived messages in memory and when you connect to a client / app, those messages get synced up to the apps. However, this is a dynamic structure based on the available memory of the device. If we opened up the ability to allocate this, it would be problematic from the perspective of different platform targets having different amount of resources available as well as that memory being potentially allocated away from other uses like the the node database, configuration, etc. Perhaps in the future, we'll be able to add functionality to offload some of this to external flash storage or other means.

Dorn8010 commented 2 months ago

I get your point, but in the current version this keeps this platform at great toy level because no consistent messaging is possible. If this structure can be implemented in the planning - since it affects the communication to all apps too - and check the MemAvail before offering it, this will ensure to make this platform usable when platforms with more memory emerge. e.g. the cheap Heltec V3 has probably enough mem for at least the last 64 msgs. This would help already a lot.

Sorry to keep you busy with my answer, but I really think this is not a comfort function but a necessary base function. This will enable the first time a reliable messaging over Meshtastic - since the device disconnects constantly from the msg device (either BT or Wifi). Otherwise the Lora works but at the end the user doesn't get reliably his messages and will leave. This happens consistently in my area, where people are really excited about the project but cannot get reliable msgs - i.e. a consistent chat.

Dorn8010 commented 2 months ago

P.S.: A simple msg counter per active chat at least informing that X msg were lost would already help.

Dorn8010 commented 2 months ago

Mem usage with

less than 16KB

Just a counter per chat with int16 would be 2 bytes per chat.

APP would just compare stored MSGcounter to current MSGcounter and the logic for the request would come from the APP. Req over MT CLI --msgs-stored-all (all sorting logic in the APP) or --msgs-stored-fromid {chatid} {msgcounter optional}

caveman99 commented 2 months ago

Sorry if i come across a bit rude here, but this is a hardware limitation that can not be overcome with the current firmware. Period. It's not as if we didn't try. There already is a queue towards the apps. If you are losing messages that may also be caused by the EU duty cycle limit.

The last post essentially describes the Store&Forward module, which is WIP.

Dorn8010 commented 2 months ago

Not rude at all. I just think your project is so great and actually really important as it is the first time a population can be connected without a central authority. This project is revolutionary. A proper logical msg and routing design (network and graph theory - not my favorite at uni :) is really important and I just give you back the feedback that I get in my local community and from my friends (they got the MT hardware and your SW from me as birthdays presents). The hardware limitations are obvious with the cheap minimal platform but it shouldn't stop from a proper logical concept and a easy reliable user experience which - if missing - is sometimes really a killer for some great opensource projects. So sorry for keeping you busy :) Keep the great work up - if I can be of any help in this process pls let me know.

P.S.: The store and forward module is still many part of the LoRa structure if I understand right. But the problem here is the logic of the comms between LoRa hardware and the apps. Also the apps between the different platforms have WIP concerning being consistent in displaying and managing information.

Dorn8010 commented 2 months ago

Sorry if i come across a bit rude here, but this is a hardware limitation that can not be overcome with the current firmware. Period. It's not as if we didn't try. There already is a queue towards the apps. If you are losing messages that may also be caused by the EU duty cycle limit.

The last post essentially describes the Store&Forward module, which is WIP.

The queue is working on the device, i.e. the device shows the last msg - but this displayed msg is never transferred to the app - not Android, not iOS not on the Web Interface. So the comms/logic between apps and device are the problem.

Dorn8010 commented 2 months ago

FYI "Neighbor Knowledge-based Rebroadcast algorithm for minimizing the routing overhead in Mobile Ad-hoc Networks" https://www.sciencedirect.com/science/article/abs/pii/S1570870518307145 Since the project uses the known neighbors, this paper maybe known already.

garthvh commented 2 months ago

Sorry if i come across a bit rude here, but this is a hardware limitation that can not be overcome with the current firmware. Period. It's not as if we didn't try. There already is a queue towards the apps. If you are losing messages that may also be caused by the EU duty cycle limit. The last post essentially describes the Store&Forward module, which is WIP.

The queue is working on the device, i.e. the device shows the last msg - but this displayed msg is never transferred to the app - not Android, not iOS not on the Web Interface. So the comms/logic between apps and device are the problem.

I have never seen this behavior, the phoneapi is very stable.

Dorn8010 commented 2 months ago

Tried it a couple of times. Same problem. Other simulation setup was to connect the device via USB and with another device via BT. The messages were totally inconsistent on both devices. A msg memory FIFO with a request to send the messages from the last msg ID on would solve this problem. Pls really rethink the msg logic. I think this is a major weak point of the project for now. Other than that - great work guys. :)

Dorn8010 commented 2 months ago

P.S. : The msgs can be stored in a simple file with a FIFO structure that is adapted to the available flash storage mem. So at least the user can decide which device to get and....the devices will get better and the code doesn't need to change anymore.