Get Python Type System to Version 1.0

[timburke]

pymomo now relies fairly heavily on a typing system for some of its functionality. The type system supports the following features:

• Annotate function arguments with a type so that the can be validated and converted from strings
• Conversion routines for each type from string
• Validation routines for each type that are particular to that type
• Formatting and printing routines for displaying the type in a nice fashion

This functionality is used currently in the following places:

• momo tool to convert command lines into python function calls
• syslog to display the typed parameters associated with each system log entry
• debug variable reading to display typed variables from the controller during operation

The type system works well but several outstanding features and refactoring would be useful before considering it stable. The isue tracks progress on these features.

• [ ] Add return value typing on annotated functions and use the formatting information from the type to print it in the momo tool rather than the current custom printer for each function that doesn't want the default str() conversion printing behavior
• [x] Add ability to inject types from other python modules dynamically so that users can define their own types and use them.
• [x] Add config file to momo tool that lets you set up the environment with your custom types and other config routines to make calls to momo less verbose
• [ ] Write more general unit tests using pyunit and include
• [ ] Split typedargs into a separate reporsitory and installable python package (momobase?) along with exceptions.
• [x] Add ability to inject types into momo tool
• [ ] Add a way to specify annotated objects that cannot be directly instantiated. Currently not annotating the init function leads to errors when displaying help information but init should not be callable because it relies on types that are not known to the type system or that are too complicated to be usefully represented as strings on the command line

[amcgee]

Agree on most of these, couple questions:

1. Is a config file the right way to go? Doesn't that make custom extensions more difficult?
2. Should typedargs and exceptions be a separated from pymomo (2 new repos) or should all of pymomo be together in a separate repo?

[timburke]

Not sure on either. For 2, we can keep it all together in a separate repository, my concern would be if it gets really big but I don't really have a preference on this. Only thing for splitting out typedargs is that it doesn't rely on anything else in pymomo so it could be a standalone package that other people could use. But I don't see a need to do that anytime soon so we could keep together.

For 1, I'm not set on a config file but here's the use case that I'm looking at. There is a significant amount of information that momo (the tool) needs to know about a module in order to be able to interact with it in a pleasing way. Currently this is:

• the module's python proxy object (now in pymomo/commander/proxy)
• the module's log statement definitions (this will be more clear after the PR but these define the types and specified formatting of all of the log entries so that momo can interpret the log data in a way other than just strings, numbers and binary blobs (now stored in the src directory of each module since it's also needed in the build process)
• the module's symbol map (needed for fetching variables from live modules via name, mapping addresses to symbols in log entries). This is currently a byproduct of the build process for pic12 modules and a .map file for pic24 that I haven't yet processed into a usable object)
• python type objects for any c firmware types that we want to be able to inspect or interact with using the momo tool.

You'll need these four bits of information from each module you want to interact with. Bit 1 could be constructed on the fly from inspecting the module's RPC endpoints but it won't be as nice as what you would get from a custom proxy object with annotated types, validation of arguments, added features and doc strings). I think it would be good to have these all stored with each module so that there are no global dependencies, i.e. the mainboard directory contains everything you need to interact with the mainboard. But this requires telling the momo tool about those proxy objects, log definitions and symbols. This can all be done with calls in the momo tool itself and currently that's what I do for the log definition files but it's verbose to have to call those functions each time you open the tool. So the idea I had was to have a .momorc file similar to .bashrc that is invoked whenever you run the momo tool and you could use to preload log definitions, symbol files and proxy objects for modules that you commonly use. This shouldn't limit customizability because there's nothing special about our modules and someone's modules that they make later on their own. Also there's no magic in the config file. It's just making momo calls you could make yourself but doing them for you so you don't have to every time.

Does this make sense or do you have a thought for how we could achieve this in a better way? I see this as a way of getting the proxy objects for the modules out of pymomo and into each module where they belong (I think) without sacrificing usability or customizability as well as allowing people the convenience of not having to load symbol files, log definitions, proxy objects and extra python types everytime they call momo but without us having to make a global inventory of these things that momo can query since I think that would limit customizability.