Generate RegBits definitions from SVD

[gzigzigzeo]

Hey!

First of all, this is amazing work, especially, the explanation of regbits principle and comparison to other approaches in the original repo. I were thinking of the same approach to register access when I found your perfect library.

I am wondering, WDYT on generating regbits definitions from SVD files? I understand that SVD are not comprehensive and such generator will require lots of manual settings, but at least it could facilitate the initial generation of header files for a specific MCU (which could be modified manually and/or fine tuned by generator settings). From what I understood, CMSIS is mostly generated in the same way.

I was also thinking of using modm-devices for gradual customisation of pin mapping: using information provided by modm we could prevent assigning wrong AFs to wrong pins, or prevent user from using pins non existent on specific MCU version/case.

I have started the research on this topic (and almost done SVD to object mapper in Ruby) and would highly appreciate your thoughts.

[thanks4opensource]

Thanks, @gzigzigzeo -- both for the kind words, and for "turning me on" to SVD files and the modm project, neither of which (to my shame) I was aware of.

I need to look more closely at modm. At first glance it seems like another layered approach hiding "bare metal" register-level development, similar to STM's HAL, NXP's MCUXpresso, LibOpenCM3 and the like. I have admittedly prejudiced/negative/opinionated objections to these approaches, as per https://github.com/thanks4opensource/regbits#Higher-level_wrappers and https://github.com/thanks4opensource/regbits_stm#comparison-with-c-implementations but need to investigate modm with a more open mind.

On the other hand, I think your idea of (semi-)automatically generating regbits templates from SVD files (thanks again!) is excellent. I've been using an ad-hoc, unpublished Python script I wrote to parse and convert CMSIS header (.h) files, but I'm sure you can imagine its limitations and error-prone output. Lots of post-hand-editing required.

The unambiguous syntax of SVD should eliminate most of those problems. However, as you point out, some semantics-level problems would likely remain. Still, almost certainly a much better starting point. Are SVD files available for a wide variety of MCUs? I found (https://www.keil.com/dd2/pack/) and (https://github.com/posborne/cmsis-svd), the latter of which provides a parser which might already implement most of a hypothetical SVD-to-regbits converter in addition to a large collection of SVDs. (I assume you probably already know about these, and other, resources.)

As is often the case, manufacturers include or eliminate important data like SVD files at will. For example, the NXP CMSIS "packs" from the Keil site -- which links them from (http://mcuxpresso.nxp.com/cmsis_pack) -- include only pre-parsed .h files, not the (I assume?) SVD "masters". Note that I'm grateful for at least the .h files -- I've had much trouble finding them for NXP processors including having to work from poor versions that contained C structs for peripheral registers but no bit definitions for the registers themselves(!!). (NXP has gotten progressively worse in this regard, seemingly trying to force use of their MCUXpresso development environment much as ST does with their CubeMX/CubeIDE.)

Hopefully some of this had been useful to you. I'm currently busy trying to finish and release another open-source project, but have great interest in this subject. I have some long-delayed updates to regbits (minor) and regbits_stm (fairly extensive) but an SVD-to-regbits converter would supersede them by a large margin.

Thanks again for your interest. Quite frankly, the regbits repositories haven't generated a lot of that (as per the Insights -> Traffic data), which is partly why I've been lazy about updating them. My belief is that a large majority of embedded developers will not consider using C++, and although their antipathy is partly justified (the academic C++ community's insistence on design patterns that are too inefficient for low-end hardware), I created regbits for my own use, and use it extensively. The C++ "haters"? It's their loss. ;) ;) Having even one more person involved in regbits would be a big plus.

[gzigzigzeo]

Hey, Mark!

I am really glad we see many things in the same way and I appreciate your interest in this idea.

modm is very interesting project, possibly, the best C++ embedded library I've ever encountered in terms of code quality. But you are absolutely correct, it is very high level. It was created to facilitate development of a robots for Eurobot competition. Every robot consists of multiple different MCUs. modm primary purpose is to make the codebase for different chips uniform.

I mentioned it because of their subproject called modm-devices (https://github.com/modm-io/modm-devices). SVD provides low-level registers descriptions. modm-devices, on the countrary, defines high-level hardware features. That include, for instance, information about RAM/FLASH size which could be used for linker/startup script generation. It is not the scope of regbits, but might be extremely useful if you would ever want to provide bare minimum project template. libopencm3, for instance, has such generator, but it maintains it's own device database.

modm-devices also has very detailed descriptions of the pin functions: like pin-to-alternate function mapping or the fact of the existence of specific physical pin on the specific MCU in specific casing. Using this information, we could implement static type checking for AFRL/AFRH registers over AF_? constants (my personal favourite fuck up :-) or even raise compile time errors if one tries to access pin missing physically.

Not sure it is very useful, but for some users it might be.

modm-devices uses cubemx as the source to generate modm-device descriptions. Check this very interesting blog post: http://blog.salkinium.com/modm-devices for the details. The most intriguing part is about grouping different hardware implementations of the STM32 peripherals over the whole set of devices. The post's conclusion is very cool too.

modm also has an interesting concept of multitasking: https://modm.io/how-modm-works/#modm-does-multitasking and they have a good bunch of external peripheral drivers implemented. They also maintain metarepo for stm32 CMSIS headers: https://github.com/modm-io/cmsis-header-stm32 (might be useful for your Py parser script).

I understand your pain of parsing .h files with Python. In fact, I was thinking about the same approach and even implemented the draft of such parser before I came up to SVD. From what I know, modm-devices parses header files to gather some information as well.

SVDs are available for many MCUs. https://github.com/posborne/cmsis-svd is probably the most reliable and comprehensive source accessible on the github. It supposed to be community-driven OSS, but I do not see a lot of activity there. Anyway, it is used as the source in a several other projects. For example, rust-embedded toolkit uses it: https://stm32-rs.github.io/

There are two problems with SVDs: as you mentioned, manufacturers are not doing great job of maintaining them. For example, official SVD files for STM32L1 family from st.com have nothing to do with reality. Second, most files from posborne repo define register groups, offests and bit ranges, but lack to define specific enumeration values.

For STM32 this problem could be solved by using patched versions from rust-embedded repo (see svd links in the page I mentioned above; by the way, I do not understand why they did not commit patches to the original repo). For other manufacturers it comes to either patching the SVDs, parsing .h or manual definition.

I think, patching/creating SVDs from scratch serves the interests of open source community the best because it provides universal MCU definition which could be used for any subsequent project in any other language.

I am not aware of the situation with NXP, but, I think, it worth researching.

Oh yeah, C++ and embedded is the holy war. Check this link: https://gist.github.com/salkinium/cc7236328a532c8c0f05f74c9ceb30a4 I have very basic German, so, I do not understand a lot of what's going on in that treads, but the fact of marking a discussions with pop corn boxes speaks for itself :-)

Frankly, I can not understand the origin of this hate. C++ certanly facilitates the development process, improves code quality, provides safety of strong type checking (as we see) and, in the same time, stays perfectly controllable in terms of code size. For me, C++ in embedded is, in the first place, a way of organizing code and just in the second an abstraction tool. No one can force me to instantiate unncessary objects or dynamically allocate memory, even if so is required to become certified C++ nerd ;-)

And I am still have not made my mind about embedded Rust. Rust is the cool language, but quite difficult to study. Rust code looks very idiomatic but it is hard to imagine the assembler code behind it. They invest in rust-embedded project but the repos traffic is quite low. Compile-time memory management features looks cool for embedded, but turns out to be useless for low memory ICs. Just out of pure interest, WDYT of Rust in embedded?

So, speaking regbits and generator.

I have the strong feeling that regbits might very well become the logical OSS drop-in replacement for stm32 LL libraries, especially, with some tooling for linker/startup script generation. LL is better than HAL, but still C, hence, ugly ;-)

Let me do the following. I'll take one specific chip: let's say it'll be STM32F030 since I have most of my pet projects done using this MCU and it does not exist in current repo yet. I'll try to generate regbits definitions for several specific periphs using patched SVD from rust-embedded (let's say gpio and rcc for starters). And then I'll get back to you and we discuss the result.

Does it sound like a plan to you? :-) @thanks4opensource

[thanks4opensource]

Good information and interesting ideas, @gzigzigzeo. Thanks for sharing them. Some of my thoughts follow ...

TL;DNR:

Does it sound like a plan to you? :-)

That sounds like an excellent plan.

More details ...

As you point out, modm-devices is somewhat orthogonal to what I was trying to do with regbits. I followed the links you provided and although I don't have time to delve deeply into it now, I found it even more interesting than on my first, cursory glance.

Check this very interesting blog post: http://blog.salkinium.com/modm-devices for the details.

That is one great read! I thought I was the only one who wrote README-like documents which contained anything more than "This is the repository." ;)

SVDs are available for many MCUs. https://github.com/posborne/cmsis-svd is probably the most reliable and comprehensive source accessible on the github. It supposed to be community-driven OSS, but I do not see a lot of activity there. Anyway, it is used as the source in a several other projects. For example, rust-embedded toolkit uses it: https://stm32-rs.github.io/

Yes, I had found the @posborne repository as I mentioned in my previous comment. Disappointing that it hasn't been updated. The rust link seems to be dead, BTW.

There are two problems with SVDs: as you mentioned, manufacturers are not doing great job of maintaining them. For example, official SVD files for STM32L1 family from st.com have nothing to do with reality. Second, most files from posborne repo define register groups, offests and bit ranges, but lack to define specific enumeration values.

Again, very disappointing (on both points).

For me, C++ in embedded is, in the first place, a way of organizing code and just in the second an abstraction tool. No one can force me to instantiate unncessary objects or dynamically allocate memory,

I like the quote I've seen before: "C++ as a better C". Or even better (immodestly) my decades-old one which I included in https://github.com/thanks4opensource/tri2b-quad4me#c_plus_plus_coders (scroll down a bit from that anchor to where I namecheck Bjarne Stroustrup and then add my own):

My favorite programming language in the C family is one of my own design. Fortunately, any decent C++ compiler will compile it without modification. I call my language "C+=0.5"

even if so is required to become certified C++ nerd ;-)

Yes, I've had to deal with many of those. ;)

And I am still have not made my mind about embedded Rust.

I haven't looked much at Rust, mostly out of inertia because between C/C++ and Python I haven't found the need for anything else. That could easily be the naivety of ignorance.

it is hard to imagine the assembler code behind it.

That's the genius of what Kernighan and Ritchie wrought with C, almost half a century ago now, and why it lives on with very few fundamental changes to this day.

Although I'm on the verge of recanting the statement I made in the "tri2b-qua4me" README linked above:

I share the current, common belief that "you can't beat the compiler" (except in very limited contexts).

My latest project does include a fair amount of ARM assembly. I discovered a could beat the compiler, or maybe more precisely that it was easier to use assembly compared to torturing the C/C++ code into forcing the compiler to do what I needed.

I have the strong feeling that regbits might very well become the logical OSS drop-in replacement for stm32 LL libraries,

I'd love to see adoption of regbits -- not so much to get my work acknowledged as to not have to do the ports myself. As you're doing now. ;)

But I can't imagine ST using it. The political issues are far too great: "Not invented here", the implicit admission that their own techniques are largely flawed, etc.

LL is better than HAL, but still C, hence, ugly ;-)

It's not so much the C-ugliness of LL that I object to, but rather the fact that it's not really particularly "low-level" at all. Trace through the obfuscated USB code, for example, and how LL calls back up into HAL which calls back to LL, recursively ad-nauseum. But at least that's better than the pseudo object orientation in HAL itself -- not that there's anything wrong with doing OO in C, but that in this case it seems to be done merely for the "nerd" factor and the implementation makes the code worse (slower, larger, harder to follow/understand/debug) than it would be with a simple procedural design.

[thanks4opensource]

@gzigzigzeo -- Sorry for the long delay, but did you ever get anywhere with generating regbits header files from SVDs? If you're still interested, check out the new beta release of svd2regbits.py and the associated README_svd2regbits.md (with all its caveats).

As always, feedback and new open issues welcome.

[gzigzigzeo]

@thanks4opensource Hey, Mark!

I am glad to hear that regbits is evolving! As usual, I am really impressed with the quality and comprehensiveness of the new README!

First of all, I should apologise for my sudden disappearance. At the time we had the original conversation, I could not have imagined how hard will this year turn out on me. :-( I wish I had a better news, but unfortunately, there is not much progress on my side :-(

However, I have a suggestion. As I understood, you have used CMSIS-SVD as the primary data source. I think, there is a better option for the STM chips.

I have already mentioned stm32-rs project. The purpose of this project is very alike to what we are trying to achieve here. It uses SVD files as a source for Rust code generation.

It consists of source SVD files taken directly from st.com plus a bunch of patches which are applied to original files using the tool called svdtools.

This patches mostly provide <enumeratedValues>, ranges and other specific values missing in original files from ST. Check the final description of CRC CR register, for example.

I am not sure if this patched versions have fixes for any of the problems you mentioned in the README, but they are definitely much more detailed compared to both original from ST and CMSIS-SVD ones. The repo itself is actively maintained.

If you decide to try it out, this gist might be helpful. It omits Rust dependencies useless for SVD generation.

I was going to use stm32-rs as a primary data source for my script.

By the way, regarding the licensing (footnote 8 and 9) - stm32-rs provides the link to the source and SLA. Seems enough, at least, for STM.

Keil provides a lots of SVDs inside chip packs as well.

I also had a closer look to modm-devices. It can be used to generate linker and startup scripts. It also has comprehensive information about the interrupt tables. WDYT, would it be useful at all? I mean, some kind of bare metal template project generator?

I thought that it could also be used for providing better semantics and constraints to the generated code (for example, AF_USART_CTS = 7 instead of AF0 = 7, which could be applied to a specific pins only). But looks like this will involve complicated svd2modm matching logic.

I have an idea that the original modm-devices generator could be rewritten to output extra SVD patches with all that supplementary information instead of custom XMLs. A single data source is always better. But this is just an idea which needs futher research.

Anyway, I have some Python code to work with existing modm files, which might be helpful if you decide to use modm-devices somehow.

Really hope that I'll be able to have a closer look to the generator and a generated code in the near future and come back with some specific feedback and/or PRs.

[thanks4opensource]

Sorry to hear about your troubles, @gzigzigzeo. 2020 sucked for just about everyone, myself included.

I finally had a chance to look more closely at modm-devices and stm32-rs. Here are my thoughts:

modm-devices is an amazing body of work. The history, effort, depth, and results are incredible. @salkinium's blog entries top even my READMEs, and I'm immodestly proud of the latter.

All that said, at least at this point in time I don't want to take regbits into that arena. I very much don't want it to become a library, an abstraction layer, a development system, etc. Before I started regbits I was writing a whole bunch of ad-hoc utilities (class Spi { public: void init(spi_typedef* spi_periph, enum master_or_slave, unsigned baud_rate ...). What I found, as I'd hoped, was that "just doing it" inline (given the ease regbits provides) was just as easy as calling the utility layers. This is my big objection to libs like STM HAL -- you still need to know the exact details of the hardware API, and the layering provides little or no abstraction/portability benefit.

stm32-rs's svdtools is also fantastic, and exactly what's needed for regbits. I have some quibbles with it, like the poorly formatted patched SVD XML output (but who cares? never need to read it anyway), commandline API (path to original SVD file hardcoded in the svdtools .yaml files), and its Python dependencies (I think bunch is the greatest thing since sliced bread but I don't use it in my distributed Python code because I don't want to require using pip installation), but they're all just that, quibbles.

I had already decided that I was going to modify the (horrible STM) SVD files instead of fixing them, post-svd2regbits, in the .hxx headers. That way, if published (in their entirety or as diff patches) they'd be of use for CMSIS converters, etc. I'd started on STM32L0X1 which is a chip series I'm using, and it wasn't that bad -- as usual I made some simple Emacs macros to take cut-and-paste from the reference manuals and convert it to SVD XML.

But the stm32-rs .yaml files already have all or almost all of what I was trying to do! They even fix the brain-damaged STM ADC channels definitions by turning them into <cluster> arrays, as they should be. Again, I quibble with choice of YAML (XKCD, "Standards"), but whatever. I hate XML, too, but I'd have kept it to match the input sources. I'll write to whatever svdtools requires, and, as I said, I might not have to write anything at all as they've done everything already.

In the near term, I need to improve svd2regbits to handle <derivedFrom> tags in register <field> definitions. This shouldn't be hard since I already have a recursive XML traverser in the code. In the long term I don't want to be doing that, instead describing multiple fields as arrays of a single definition. (This is the major upcoming regbits improvement.) It's also why I quickly abandoned the existing SVD parsers I found and started with, as they purposely traverse and flatten nested definitions and at every level (TIM1 and TIM8 are both AdvancedTimers and shouldn't be two separate/identical structs/classes) I want to preserve the "derivations".

Hope some of this matches your needs. Thanks again for the continued interest.

[salkinium]

Ohai, thanks for all the kind words!

I've been down this road too, however, I've decided to use the CMSIS Headers from ST for modm for two main reasons:

1. The memory map data differs slightly between CMSIS header files, SVD files and PDF Reference Manual. This is pretty bad as it shows that ST has no single representation in place or it does not have a data pipeline in place to keep all of them in sync. Of these three sources, only the CMSIS header are widely validated via compilation in actual source code, thus it'll be the most accurate and most up-to-date.

2. Third-party code will use the CMSIS Header definitions, since they are the de-facto standard, so we have no choice but to provide/use these headers anyways. Nobody escapes the clutches of ST!1!!

We've also implemented our own C++ Register abstraction via enum classes for using as the register map for external devices, but I've long thought that we should instead build a small SVD file and then code gen the C++ code for it using a library like this. But my ToDo list is so long already…

Anyways, if you want to parse the CMSIS Header files, I've implemented a GCC-based approach, that delegates the macro resolution to GCC and "unflattens" the defines into a peripheral structure. Here's a dump of the resulting data in a rather useless debug format: https://github.com/salkinium/stm32-memory I wanted to use this data to algorithmically categorize the peripheral type by matching memory maps into the smallest possible set, but never got there due to work/uni…

Perhaps this is more useful for you, feel free to extract the code and make it your own.

[thanks4opensource]

Thanks, @salkinium. I understand your rationale for using the CMSIS headers as the prime/definitive source. (The discrepancies between ST's CMSIS headers, SVDs, HAL and LL headers, reference manuals, datasheets, application notes, etc, etc never fail to amaze me.) I started by hand-generating regbits .hxx files like the ones in this repository directly from CMSIS. I'd at times thought about using the C preprocessor to extract the #define information, although as I've mentioned before I try to avoid flattening in general -- and I also don't have your amazing file parsing skills.;) It would be interesting to turn stm_header.py into a full CMSIS-to-SVD converter.

All that said, for right now I want to stay with SVDs. My bottom-line opinion is that vendors should provide accurate SVDs for their products. I don't know what ARM's policy for their licensees is (they probably don't care as long as the royalties are paid) but I wish they'd pressure them to do so. I'm also interested in non-ST chips and/or non-ARM architectures. The RISC-V people seem to be doing something with SVDs and/or xsvd, the latter which seems to be an improvement (addressing many of my exact complaints about SVD) although I'd be reticent about jumping to Yet Another Standard.

So, again, for now my plan is to use svdtools with the excellent stm32-rs patches, and/or contribute my own fixes as I find them. That hypothetical CMSIS-to-SVD converter sure would be nice as an alternate/merge path, though.:) As always, I don't expect much help from ST -- they're completely committed to their Cube/HAL dev environment (that's where the enumerated values are hidden) and the customer lock-in that provides them.:(

[gzigzigzeo]

Hey.

I think, CMSIS-to-SVD converter might definitely be a way, but I am not sure that a potential effort of creating it would worth a potential benefits. stm32-rs is not covering 100% of registers yet. In fact, for some families it covers less than 50%. In the another hand, it is community supported, so, the coverage is growing. It is also in a way proven by real world rust projects, as like as CMSIS. Providing patches to stm32-rs project would be possible easier in terms of development and surely beneficial for the community. Again, in the another hand, SVDs generated from CMSIS would have 99% coverage onset. And possible will inherit all errors from a source and require patching. Lot to consider :-)

@thanks4opensource I am sorry if I get you wrong, but I have the objection about your points on stm32-rs. It has cli interface which requires several Python dependencies, yes. But I do not see it as a big problem. I believe, you are not supposed to generate every SVD file manually. It has the script which does the job for all files at the same time, and this script could be run once (or periodically, on CI) in isolated environment (like, in Docker).

I also prefer to have as less dependencies as I can, but I do not see having pip deps as a big problem for svd2regbits. First of all, I think, that an end user would use the generated version of regbits.hpp file for his specific chip. I do not think that there are a lot of scenarios where the generator would be run manually by an end user. So, I think, having dependencies (in requirements.txt, for instance) is totally fine here. I also think, that some tool like dip could be used facilitate local development. In any case, there are no strong guarantees that a user has the required python3 version on his local development machine, so, python dependencies would not make things worse :-)

[thanks4opensource]

Yes, @gzigzigzeo , there's always lots to think about with this. From the very beginning of regbits I've struggled with how much I want to match ST's documentation/CMSIS/HAL (and now SVDs) and how much I'm willing to diverge for something "better". A big example is arrays (SVD's [%s] construct) of register <field>s. This is something I've wanted to put into regbits for a long time, so ST-style constants (using the crazy old STM32F1 series GPIO configuration as an example) like this:

gpioa->crh /= Gpio::Crl::CNF5_ALTFUNC_OPEN_DRAIN | Gpio::Crl::MODE5_OUTPUT_2_MHZ;

would turn into this:

gpioa->crh /= Gpio::Crl::CNF<5, Gpio::Crl::Cnf::ALTFUNC_OPEN_DRAIN)>() | Gpio::Crl::CNF<5, Gpio::Crl::Mode::OUTPUT_2_MHZ>();

(The regbits template functions make this completely safe against Gpio::Crl::CNF<8 ..., etc). Doing it this new way would allow, at runtime:

unsigned port_number)
{
    if (Gpio::Crl::cnf_valid(port_number))  // no need to check Cpio::Crl::mode_valid(), is same range
        gpioa->crh /= Gpio::Crl::cnf<Gpio::Crl::Cnf::ALTFUNC_OPEN_DRAIN>(port_number) | Cpio::Crl::mode<Gpio::Crl::Mode::OUTPUT_2_MHZ>(port_number);

without any need for a switch/case block to choose the correct bit <field> values.

But it would, of course, require a significant change to the SVD or the stm32-rs YAML (or post-editing of the svd2regbits.py output, which I've already rejected as a workable option).

Regarding dependencies, yes, either you got me wrong or, more likely, I wasn't clear enough. I was trying to say that my objections were minor. I try to avoid dependencies when possible, but understand that sometimes (as with svdtools) they are needed (YAML and others). Hey: My svd2regbits.py needs an XML module, although it tries to find one of XML, LXML, etc at runtime and so far works with all of them (this may change -- XML's Xpath support stinks, and I may have to require LMXL).

And I'm very "old school" when it comes to Docker, etc. I like to keep things as simple as possible (yes, I know that in some ways this is exactly what Docker is trying to do). Note that even the svdtools make example from their website requires venv which I don't have on my system (and of course pip didn't install). Not a problem -- I worked around it, and in any case the svdtools developers can design and distribute whatever they want. As you've pointed out before, stm32-rs deserves great credit for making the YAML files usable outside of Rust development. BTW, I was dragged kicking and screaming away from Python2, but with the end of support it was time. It does seem that minor version inconsistencies with Python3 are worse than they were with 2 (3.6 is often required). I've been wanting what became the "walrus operator" in Python forever, but I don't even have the minor version where it came in on my system. And likewise C++ 14/17/21 (11 fixed many of the things that were completely broken in 03 and 08, and is almost a decade old now, so I'm fine with that).

As always, I greatly appreciate having someone I can discuss issues at this deep level with.