Implement board-specific build actions

[PMunch]

One thing I want to implement for Ratel is board-specific build actions and other actions such as getting the size of the various sections of the binary, and uploading the binary to the board. The goal is to be able to just clone some repo and then run ratel upload /dev/ttyUSB0 for example and have it build and upload the code to the selected port, based on the board that was selected in the config file (maybe also support an override option). The idea is to allow board integration maintainers to write nice scripts which hide the hairy logic from users, similar to what is done in the Arduino IDE, and make switching boards super easy. But I have some different ideas for how this could be achieved:

Option 1 - Simple bash scripts

Essentially write the ratel program as simple wrapper which extracts the currently selected board, and then finds the given sub command (in the above example build) and then runs a bash script named build in the boards specified folder. Pros: Easy to implement, easy to simply paste the command you need into a bash script Cons: Only works on Linux (could work on more platforms through multiple scripts though), parsing of arguments should probably be done within Ratel itself to avoid bash scripts from doing the parsing differently and breaking the easy-board-switching goal.

Option 2 - NimScript

The ratel program would perform the same work as above, but instead of running shell scripts would call out to NimScript. In the beginning this could be a simple nim e implementation, but it could be extended to use nimscripter in order to create a more extensive toolkit for board integrators to use. Pros: Can build a tighter ecosystem where the scripts can query for more information through procedures, easier versioning, all in all better control (could even build our own executor and have a --dry-run option which didn't run external programs). Cons: Would still need to call out to external programs for the actual work, all scripts might end up just becoming a single execute call.

I want your feedback! Do you have any other bright ideas?

[yglukhov]

I was thinking something along these lines.

• User imports board config into config.nims
• Board config provides upload function onBoardUpload() do(path: string, port: string): ...
• It can also provide other functions like findPortsWithConnectedDevices() do() -> seq[string]: ..., so that user can just do ratel upload without specifying the port.
• Calling ratel upload /dev/... will eventually call this function.

The implementation of such system should be relatively straightforward too, there should be a nims file with all the glue logic and import project config.nims in front.

[auxym]

Definitely not bash :)

I agree with Option 2 - nimscripts, mostly. The "shelling out" con to me isn't too much of a problem, it's sort of expected that you'll need to call avrdude or bossa or OpenOCD, etc to flash the code, anyways.

How do you see it implemented? A single nimscript file per function, eg. flash.nims, the you call nim -e on it? Or does a single script have all the functionality in it with a common CLI contract that Ratel understands?

An alternative might a nim module that gets compiled on the fly? One advantage would be better support for subprocesses, support for which is limited in nimscript.

[PMunch]

I've been playing around with what @yglukhov suggested. Essentially I have implemented a small program using nimscripter which reads the config.nims file of the project, which then can contain tasks like build, upload, size, etc. which the program can call. Since nimscripter (or anything interfacing with the compiler) is free to implement logic in Nim code and expose it as functions for the running module I don't worry too much about limited capabilities, if we need something in particular it would be easy to add it in. Currently I use nimscript.nims to allow the switch and -- statements to be there, but I believe a better way of doing it would be to shim these so that we can read the options from a simple table instead (don't want --os:any to stop us from using os features in our scrips now do we?). Essentially this allows us much greater control than simply running bash scripts or compiling programs on the fly, we can provide a lot more features since we would be in the chain of execution.

I believe this is the right way to go about doing this. I'm implementing the simple things I have in my makefiles for now, and hopefully I'll get findPortsWithConnectedDevices working as well (although that turned out to be a bit tricky, especially for devices like the Teensy 2.0 which doesn't show up until you reboot it).

[yglukhov]

nimscripter

This sounds harder than I anticipated though. Why not just nim e?

[PMunch]

Mostly just to be able to run different procedures from within the script. As far as I know using nim e it will just run the file as is, so trying to run config.nims wouldn't be able to run anything but the top-level code. Of course it would be possible to simply read in the script, concatenate some things, and then run nim e on that, but this seems like too much of a hack for my liking. With nimscripter a command like ratel build could be translated as "load config.nims, and run the build procedure in that script". This allows the ratel command to have more control over how flags are parsed, how options are treated, and in general keep the scripts behaving more similarly without too much hackery.

[yglukhov]

Of course it would be possible to simply read in the script, concatenate some things

Yes, that's very close to how I thought to do it. I don't have a strong opinion about your approach vs my, but for the sake of completeness, here's how it could work in a least hacky way with nim e.

ratel cmdline tool calls out to nim e -d:project_root ratelcmdline.nims $@, ratelcmdline.nims imports project_root/config.nims, does the cmdline-parsing, and dispatches to appropriate board hooks. It doesn't look overly hacky to me, no code concatenation :), although architecturally it might seem ... unconventional? :) Using nimscripter might have a potential benefit (yet to be proven) of providing natively-compiled api to the scripts, but avoiding nimscripter has a benefit of not embedding particular nim version into ratel that might differ with actual nim version.

But again, feel free to do it however you like.

[PMunch]

Aah I see what you mean, so all the Ratel logic would be shared through that one script which would then import and call from the board configuration. That makes sense, and it means that Ratel doesn't have to embed the whole compiler (granted it's only a couple of megabytes, but still). Downside is as you mention that we can't implement native logic, and parsing the switches would have to be a bit of a hack since we can't use the built in switch and -- procs amongst others.

I'll have to think about this, it's late here now but hopefully I've come up with a path forward tomorrow.

[PMunch]

I implemented this in https://github.com/PMunch/ratel/commit/055530ece96ecc661fb6d4f39cdb182ce569fff8, see what you think! Feedback appreciated!