Make generated code configurable without re-compiling

[brice-morin]

For the generation of config files, I suggest we right now use those optional annotations on properties:

• @saves "on change"
• @saves "on exit"

By default, we do not save properties, just use the initial value provided in the config file. "on change" saves the value of the properties every time it is changed through a proper ThingML assignment (i.e. it will most likely not be saved if updated through an extern statement). "on exit" only saves the properties when the program terminates (typically on Ctrl+C).

@ffleurey, @jakhog As much as possible, only rely on those annotations when implementing this feature in the compilers and in helpers (I already started updating some helpers).

We might consider promoting those annotations at some point to proper concepts.

[brice-morin]

Also we should decide what to do with non-annotated properties. Should they be in the configuration file? It might be convenient to be able to start a thingml-generated with alternative initial values without recompiling the thingml and the generated code. On the other hand, it introduces some overhead to have the configuration file, and read it.

We could use this annotation:

• @saves "manual"

Then we could ignore all non-annotated properties.

[brice-morin]

More generally, it would be great if (part of) the config file could be overridden by environment variables, which can themselves be overridden by command line arguments. This way, we can have quite a flexible configuration solution when we start a program, like most would be defined in a file, some as env variables and properties we want to change quickly without changing the configuration file can be passed as arguments.

[jakhog]

Hmm, I'm not quite sure if I like the idea of loading all possible properties given in a config file. I feel like there should be an explicit annotation (or later keyword) that shows that a given property is allowed to be initialized at run-time (as opposed to compile-time which is what we have now).

In my opinion, the @saves annotation should be changed to @persistence, and could have the value "on change", "on exit", "manual". To be honest, I'm not even sure how easy it is to implement the "on exit" one? Maybe we should just support "on change" and "manual", and leave it up to the user to save it in an on exit action?

In response to the override of values from your last comment, I think we could be even more generic and have some kind of priority notion of the configuration plugins, so that a plugin with a higher priority will be able to override the values changed by the lower priority plugins.

[jakhog]

Another thing that would be useful would be to be able to run a command to generate a template configuration file with the proper names of the properties that can be set in the file, and possibly their default values (as defined in the ThingML) code.

I'm not sure if this functionality should be in the ThingML compiler, or in the generated executable file? Is this type of behavior something that we already have for plugins? That they can define pieces of code that can be invoked from the compilers? Like an argument on the CLI-compiler registry?

Or maybe this is another compiler?

[jakhog]

Also, we should also consider sessions. How do we want to deal with persistent properties when they are used together with sessions?

[brice-morin]

Well, I do not think we should make sessions persistent. They "inherit" their values at fork time from the root (which can be persistent), and the sessions are "freed" when they reach their final state (or the whole program terminates).

As for "on exit" behavior, we typically have a hook on SIGINT/SIGTERM or whatever signal is used to kill a program. It is just a matter of calling a "set" on all persistent properties and maybe a "save" in the API on the config lib. Quite easy. Maybe not on Arduino, where libconfuse or similar is probably overkill.

@persistent is more accurate, but IMHO too verbose for an annotation. Would work as a keyword with completion, though.

[brice-morin]

Or "persist" would be quite good. Persist on change, persist on exit, etc actually means something.

[brice-morin]

As for the template, once a property is persistent, we should set it from the config file (or DB, etc), so we have to generate this template, as we compile the ThingML file. The main will then read the file (which may already have been updated) as we run the program for the first time. Or we can generate some complex behavior to check if the file there, if not use default values, it it is there, check that each individual property is there, etc. Too complex. If we make a property persistent, we should configure it from the file from the very first time. Should be easy to generate a file, or populate a DB with the default values we used to put direct in the main.

[brice-morin]

Though we could also generate a simple main that generate the file or populate DB, so as we do not need to call the ThingML compilers, but can generate e.g. a 100% C project to configure and run programs, which might be easier to embed in some tool chains.

[brice-morin]

Or use a program (generated by ThingML or not) to populate the file/db. The point is the main should assume a complete configuration file/db when it starts, I would think.

[brice-morin]

Just some notes for later.

Here is a proposal to try to decouple the configuration plugins from the "main" plugin. In the generate main, for whatever language (here :

//section to declare the properties for each instance, by default initialized with the initial values computed from ThingML

var myInstance_i : Integer = 0;
var myInstance_j : Boolean = false;
var myInstance_k : String= "hello";

//each configuration plugin (from least-to-most priority) can override value
//Plugin 1
myInstance_i = readFromFile("myInstance", "i");
myInstance_j = readFromFile("myInstance", "j");  

//Plugin 2
myInstance_j = readFromDB("myInstance", "i");  

//Then we can create instances
myInstance = new MyInstance(i, j, k);
//i will be set from the DB
//j will be set from file
//k will be set from ThingML

That way we should be able to handle multiple configuration plugins