Roadmap discussions - Githubissues

Hi there. I couldn't see such a discussion in progress, and the roadmap says "GitHub issues are a great place to discuss ideas." So here we are. I think the easiest way to begin what I genuinely hope can be a fruitful discussion is to run through the current road map and add my comments (italicised).

I'd like to emphasise that while some of these remarks might seem critical, they are not intended in any way to be an attack on the existing CLI or those who wrote and maintain it. I have a lot of experience of software development and things usually are the way they are for reasons, so over the years I have learned not to judge. In using terms like "low quality" I merely seek to use concise descriptions, not to cause personal hurt to anyone in a project that, whatever else can be said about it, has brought medium-range mesh technology to a considerable audience already. So you might consider these notes as suggestions for coping with the next wave of users, who will typically be greater in number and perhaps less technically able.

Also, please bear in mind that I'm relatively new to Meshtastic so I have little knowledge of the history, and do not by any means consider myself expert. So please forgive any blunders due to simple ignorance.

Types
- type annotations throughout the codebase - absolutely agree, but I don't think this should be the team's first priority given the generally low current software quality.
- mypy running in CI to type-check new code - again desirable, but the principal issues to be solved in the current client are primarily structural rather than datatype mismatches.
async-friendliness - I'm not sure what this means. The code is a monolith, so it isn't immediately obvious to me which pieces run in parallel and how async will improve things (though I'm quite prepared to believe it would). So a little more detail here might clarify.
CLI completeness & consistency
- the CLI should support all features of the firmware - naturally!
- there should be a consistent output format available for shell scripting - this is currently one of the more frustrating aspects of the CLI, which would have been better designed to separate the radio interface from presentation tasks in a layer which could then provide the data to generate output in textual or JSON format.
CLI input validation & documentation
- what arguments and options are compatible & incompatible with one another? - The structure of the command line needs to be modified, because representing arguments and options exclusively using options (whether single- or double-dashed) has created these problems unnecessarily. Consider restructuring to have the CLI respond to different subcommands such as info, set and get, each of which can then more easily allow only the arguments and options that make sense. Subcommands can in turn also have their own sub-commands, allowing the command line to express operations in a relatively comprehensible way. The click package is one way to create such CLIs. See programs like docker, for example, to better understand the advantages of this approach.
- can the options be restructured in a way that is more self-documenting? - see above.
- pubsub events should be documented clearly - Probably most things should, but in this instance a high-level description of the feature would be a helpful start. The current pubsub interface is probably the closest thing to a foundation SDK (see below) I've come across so far.
helpers for third-party code
- it should be easy to write a script that supports similar options to the CLI so many tools support the same ways of connecting to nodes - What you appear to describe here is a software development kit (SDK) - the very component that I suggest above under "CLI completeness & consistency" that could simplify the CLI architecture can also, if made available as a component, allow other developers to build Meshtastic applications without the hard work of writing lower-level support code. This would encourage a base of developers familiar with Meshtastic, promoting further growth in the ecosystem.
interactive client - Interactive in what way? If it's going to interact with the user we need to specify how.
data storage & processing
- there should be a standardized way of recording packets for later use, debugging, etc. - Such a standard already exists in the form of pcap. By adopting pcap you would also be adopting compatibility with an existing ecosystem including tools like Wireshark, which can decode packets captured in real-time.
- a sqlite database schema and tools for writing to it may be a good starting point - This seems like premature "solutionising" - sure, SQLite is good at storing things, but that doesn't necessarily mean it's what you need here. Better to start by considering the requirements ...
- enable maps, charts, visualizations - maps and charts are always going to be tricky in a command-line program.

There are a number of broader issues that, if not addressed, will hinder the growth of the architecture and the developer community. As this commentary is already over-long I mention only those which come most readily to mind.

Because of the lack of a clear conceptual architecture, the CLI as currently implemented is less easy than it might be to understand, use, maintain and extend.
Testing is being done at the wrong places much of the time: rather than testing that a specific interaction with a component generates a specific exception, many tests set up an error condition then run the whole command-line program and examine the error messages generated by the application code. This is far too complex for unit tests.
At present the code is quite fragile, and changes in one area can surface unexpected issues in unrelated areas.
Exceptions are frequently not used to notify of error conditions. Instead the code often attempts to respond appropriately to events at far too low a level, where essential applications context is unavailable to permit more directed responses.

Thank you for reading this message. I do not believe any of the issues I have mentioned are insoluble, but a piecemeal approach definitely won't effect the necessary transformation. I'll be happy to take part in further discussions about the CLI if that would be regarded as helpful.

Hi, thanks for the detailed thoughts! I agree with a lot of what you're saying, and to be honest that roadmap was something I threw together pretty quickly to hopefully stimulate interest and discussion, so I'm glad you looked it over. I've only been the maintainer for a couple of months myself and this library & CLI was without a maintainer for a while, so there's definitely a lot of potential things to improve. I'll go through some of the stuff you've said and add my own thoughts (sorry in advance for my lack of concision!):

Types:

I agree that adding types to the existing code isn't terribly high-priority, but I figure it's good practice to be progressively adding types as stuff gets changed
mypy is actually running in CI now and I should probably remove or amend that bit in the README. We're not running it on very strict settings, but we've at least got something there to progressively improve on. Part of my philosophy around something like this that doesn't have many developers or time to be devoted to it is to have some things in place that nudge each contribution in a better direction. Not a complete solution to anything, but something to hopefully provide benefits in practice given the constraints. In any case, I'm sure having it running isn't hurting!

Async-friendliness: This one was a little deliberately vague on my part. This library largely predates widespread use of Python's async/await functionality, and I think that it should be set up in a way that makes it easy to use in that context, but I'm also not terribly experienced with that myself. I think that this point would need some assessment from someone who knows more than I do, or for me to learn more. It could be that it's already in a perfectly fine place and I just don't know it!

Completeness/consistency: This is definitely some of my top priority stuff here, particularly a more nicely machine-readable output. I've been devoting some thought (as I can) to the best structure for that output, but I suspect the answer is probably to do as little transformation of the raw protocol buffers as we can, which may also probably require some internal restructuring. This also ties into other sections of the roadmap around restructuring of the CLI and around persistence, of course. Ultimately, what I'd probably like is to have some defined structure that can represent everything we can return in as close a format to the raw protocol buffers as is reasonable, and have various subcommands/options/etc. simply decide what parts to populate.

Input validation/documentation/restructuring: I agree that using subcommands is the way forward; I also think we need to keep the old options for some amount of time, which is what probably most complicates the process of adding better argument validation. I think that the way to do this may be to move things to a subcommand structure and reimplement the existing options in terms of the new structure. I have not (yet) sat down to think of an appropriate collection of subcommands (and sub-subcommands, etc.) but would welcome help on it; I'll plan to make an issue for discussion of that topic soon, because I think it's a pretty well-defined thing to figure out and definitely should include more voices than mine if possible! I think that like the machine-readable output, this is another place where sticking close to the protocol buffers would probably ease future maintainability. (somewhat of an aside, but I will also admit that I'm not a huge fan of click myself, because I find argparse more accessible, but I'm not necessarily opposed to using that either. argparse does support a subcommand-type structure as well, so the particular choice of CLI-building tool is probably secondary to planning out the desired structure anyway)

third-party helpers/SDK: yes, this is pretty much what I wanted here. The particular small thing that was the impetus for me including this in the list was that most quick scripts people throw together tend to have a hardcoded SerialInterface that they use, which is really a shame given that it would require people to edit the scripts to use alternate ports or the TCP/BLE interfaces. I've shied away from using the term SDK since most of the SDKs I've interacted with have been more obstructive & opinionated than helpful, but the goal is perhaps the same. I think that creating a more careful layered/composable system within this project will be the biggest part of doing this, anyway -- if this project can be built as a base library with layers for things like common CLI arguments and particular CLI subcommands, all of those things become more reusable by others too.

Interactive client: This is definitely a low priority for me, especially within this repository itself. I'm thinking here mostly of something like https://github.com/pdxlocations/meshtastic-curses-client (which uses this library!) or https://github.com/PeterGrace/meshtui which uses the Rust library. This entry in the roadmap predates both of those projects, though. I'm happy to have that sort of project stay there for now, so I'll probably take this entry out of the roadmap.

Data storage/processing: I think my explanation of what I was thinking of here is lacking, and a better term to use is probably "persistence". We should have something that allows the library (and therefore also the CLI) to retain information on-disk for a variety of later uses, including things like debugging but also things like message history, maps & charts, mesh analysis, etc -- and most of these things would be in other tools and projects, not directly within this library. For my own purposes, what I want this for is so I can build something to analyze my own local mesh, to inform decisions like where to place new nodes, changes to node roles to improve routing, most common paths through the mesh, and so on. I'd rather do it from a solid base than invent my own persistence layer. Pcap isn't an unreasonable choice for debugging since, as you point out, it could be made to work with tools like Wireshark, but it's less suitable as an application persistence layer. SQLite was also just an idea and not one I'm devoted to by any means; there's a user in discord who I believe is working on a proposal built around storing data in BSON format and I'm looking forward to seeing ideas there.

Tests: yeah, the tests need a lot of work. I've barely touched them other than to get them back to passing to hopefully prevent some amount of regression. I've been considering using hypothesis to test a lot of permutations of the lowest-level stuff, but many of the refactors that both of us want would help test different bits more in isolation.

Exceptions: I agree, and I think this will be an important part of decoupling the CLI/presentation parts of the codebase from the library parts of the codebase.

In short/summary, I think you and I agree on most of these things and I'd look forward to your contributions or ideas. I'll try to update the roadmap for some things that have changed in the last few months and make some github issues for specific ideas or discussions that should happen (and you should feel free to do the same).

For now, gonna get back to the day job. Thanks again for your thoughts!

Hi again. Rather than trying to keep this thread discussing everything, everywhere, all at once, might it be more fruitful to use it as a birthing suite for directed issues to start fixing things up?

In that spirit I'm ignoring the stuff either you've already done or we agree aren't high priority, and replying as concisely as I can to the rest. Anything else can stay on the roadmap (editing which could be another issue!)

Async-friendliness some libraries have adopted architectures that allow sync and async to share common source code - one that comes to mind is Textual. This probably needs a longer discussion.

Completeness/consistency agree about doing as little as possible with the raw protocol buffers, but doesn't there need to be a coherent layer above that? I haven't seen a sensible mapping of the Meshtastic architecture to the Internet or ISO layering models, and I'm still struggling to understand packet flows across the mesh. This layer, which for now I'll call the Packet SDK (PSDK) layer, could form the basis of the SDK. Would it be possible to use the existing pubsub events to initially built that layer? If so, we could then rebuild the RF drivers without affecting the PSDK layer at all, adding a much-needed isolation to parts of the code. (So much for conciseness).

Input validation/documentation/restructuring I have similar feelings, but did one day beging to try and noodle things into families, with the following result. Clearly much to do yet (I don't know what half the options do), but it may give you a starting point. Is this enough to start an issue?

mt bluetooth enable | disable
mt bluetooth pairing random | none | N
mt bluetooth uplink | downlink enable | disable

mt channel add --number N --name NAME --encryption none | default | random | simple N | psk PSK --uplink on | off --downlink on | off | position_precision PRECISION
mt channel set --number N --name NAME --encryption none | default | random | simple N | psk PSK --uplink on | off --downlink on | off | position_precision PRECISION
mt channel remove N

mt device role ROLE
mt device rebroadcast ALL | ALL_SKIP_DECODING | LOCAL_ONLY | KNOWN_ONLY
mt device serial-console on | off
mt device debug-log on | off
mt device user-gpio N
mt device buzzer-gpio N
mt device broadcast-interval N
mt device double-tap on | off
mt device managed

mt display duration N
mt display carousel N
mt display compass on | off
mt display gps DEC | DMS | UTM | MGRS | OLC | OSGR
mt display metric | imperial
mt display flip
mt display oled AUTO | SSD1306 | SH1106 | SH1107
mt display mode DEFAULT | COLOUR | INVERTED | TWOCOLOR
mt display bold on | off
mt display tap_to-wake on | off

As to your dislike for click I wasn't advocating for it, it's simply the CLI library I know best, but I'll happily work with anything sensible that does the job.

third-party helpers/SDK agree, the restructuring (or at least the mapping out of a consistent conceptual framework for it) is the most important thing to get right, and therefore probably one of the more significant upcoming tasks.

Interactive client yes, the most significant thing we can do to multiply our efforts is offer real help to people who are building sensible third-party tools.

Data storage/processing Fine. Ideally define an abstract storage layer and then provide drivers for the various different real stores. Even if there's only one driver initially, it will help to focus on separation between the storage and its client(s).

In short/summary glad we appear to see eye-to-eye. Since we both have day jobs I'm not looking for any overnight transformations, but with some sensible engineering we can get the existing code into better shape, and create building blocks of general utility. Thanks again for your positive responses.

I agree on splitting things out. I'm working on creating a few issues and I'll make sure to link them back here as I make them.

If I may make a suggestion on Data storage/processing, I would prefer to keep things really dirt simple and easy consume. Something like a sqlite database makes a lot of since because it's portable, easy to query with any language, and there's tons of tooling around it.

Good point. Everything you say is true. Remember, though, that relational storage isn't a natural way to store recursive/multilevel composite data structures, though obviously it can be done. I certainly wouldn't want to exclude people who wanted to use SQLite. Some thought is required as to what gets stored, whether there's an easy way for users to modify what gets stored, and how people consume the stored data. Ideally there'd be a range of storage backends available for different use cases, each using a common interface to the radio data.

Another thought: rather than proceed encumbered (?) by the existing architecture, would it make sense to, for example, write a utility to maintain the settings in a file, and modify the client to load its settings from the external file when instructed to do so by a (e.g.) load command?

--set/get would then go, to be performed outside the client by the posited utility. It would clearly need the ability to load the configuration, modify it and store it back. Its development would therefore allow safe experimentation with suitable Python structures to better represent the configuration.

This approach offers the further possibility of maintaining configuration data under source code control. Of course it could be that it's already possible and I just don't know enough.

The file-based settings stuff is basically what --configure and --export-config are set up to do now, but it doesn't take much time in the issues section to find weaknesses with their current implementations. I don't think I'd like moving all the configuration to that model, especially since the firmware isn't really set up for it (every sort of full export/full import of config is built around just doing a bunch of separate admin commands), and that can get heavyhanded quickly, especially for remote admin. But I think we should certainly have a better-functioning thing than our current commands!

This clearly isn't something either of us can find much time for right now.

meshtastic / python

Roadmap discussions #566