rust-embedded / wg

Coordination repository of the embedded devices Working Group
1.89k stars 98 forks source link

Maintenance of core parts of the embedded / Cortex-M ecosystem #46

Closed japaric closed 6 years ago

japaric commented 6 years ago

A.K.A. @japaric is a bottleneck for development :-)

There are critical components of the embedded / Cortex-M ecosystem that are currently being maintained by me. Unfortunately I don't have the bandwidth to keep up with everything so issues and PRs have been piling up in some repos.

This issue is a call for maintainers (if you want to help leave a comment!) and a place to discuss how to organize maintainership duties.

First, these are the crates that should be maintained by more people:

Abstractions for things like Serial, SPI, I2C, etc. to build generic drivers that work with AVR, ARM Cortex-M, MSP430, RISCV, etc. microcontrollers.

There's a bunch to do here as some abstractions are missing and async support is currently nonexistent. Most of the work to do here is ensuring that the established process for adding new traits is followed and that discussions move towards consensus.

Quite a few people have volunteered to help and are now collaborators so I think this one is well covered for now.

Tool used to transform CMSIS SVD files (XML files that describe the hardware registers of a device) into code for low level manipulation of hardware registers. Although the SVD spec was created by ARM svd2rust is being used for both ARM Cortex-M and MSP430 devices. In the later case TI provides DSLite files (their version of SVD files) and those are converted to SVD files using the dslite2svd tool.

It's expected that svd2rust will also grow AVR support -- Atmel has its own description file format: atdf -- using the atdf2svd approach. And there are XSVD (JSON version of SVD files) files for RISCV -- I don't think they are official though? -- so svd2rust could also grow support for RISCV via a xsvd2svd tool.

Unfortunately svd2rust has grown into an unmaintainable mess. The test suite takes a lot of time to run (it takes longer than the CI limit) and it only checks if the generated code still compiles -- it provides no feedback about whether the generated code exposes a certain interface or not. The underlying parser lacks error handling and simply panics on errors. Also, svd2rust doesn't give you helpful error messages when your SVD file has some problem -- this is an important feature to have when manually editing SVD files.

So both svd2rust and the underlying svd parser need to be refactor (read: rewritten) to make them more modular and testable. I think the approach to take here is a refactor like the one proposed in japaric/svd#37. Basically we should be able to test code generation on parts of a SVD file. For example that <enumeratedValues>..</enumeratedValues> produces this enum Foo { .. } and this API impl Foo { fn is_variant(&self) -> bool { .. } }, etc. This will let us write unit tests that can replace the huge existing tests that only check for compilation errors. And while you are at it you can add error handling to the parser and nice error messages to svd2rust.

This is a huge task but it will make it easier to continue the development of svd2rust. There a bunch of open issues related to SVD files not being supported; the refactor should make it easier to fix those issues.

If you want to tackle this let me know in the comments!


Then we have core cortex-m crates.

This repo provides access to Cortex-M specific features like access to registers, safe wrappers around assembly instructions and access to core peripherals.

One old issue in this repo is providing a svd2rust generated API to access the core peripherals. The current API has been manually written and is not as type safe as a svd2rust one would be. Fixing this requires writing or putting together an SVD file that contains only core peripherals, running svd2rust over it and committing the code to the cortex-m repo.

This crate takes care of communicating the memory layout of a program to the linker and providing implementations of language items that are required to build a no_std program.

There are a few missing features in this crate: support for placing functions / interrupt handlers / the whole program in RAM, relocation of the vector table, etc. that need a design. It may be useful to coordinate with the representatives of the other embedded targets to see if we can provide a similar user interface for these features.

This crate is used as a Cargo project template for writing Cortex-M applications.

Every now and then a user figures a new way to misfollow the instructions and encounters a new error message. These errors should be documented in the troubleshooting guide.


Regarding organization I think it may make sense to move these core components under the rust-embedded org and then create, say, a cortex-m team in charge of maintaining the cortex-m crates. As the AVR, MSP430 and RISCV communities grow they may also want to move core components into the rust-embedded org and create their own maintainer teams. Thoughts on this?

cc @ryankurte @Emilgardis might be interested in picking up svd2rust development cc @dylanmckay (AVR) @pftbest (MSP430) @dvc94ch (RISCV) see last paragraph, also svd2rust support

ryankurte commented 6 years ago

I like the idea of moving components into rust-embedded (and increasing the bus factor with that org 😂).

I'd like to see a code of conduct (something line the contributor covenant and the contacts and policy to support it, as well as contribution guidelines (ie. how to pr, policy on breaking changes and versioning, adoption of projects by rust-embedded etc.) at the org level.

Also willing to pick up some of rust2svd/svd-rs(/embedded-hal) development/support/maintenance, and to update japaric/svd#37 to get it merged as a first step in the refactor if that's useful.

dylanmckay commented 6 years ago

@japaric

It's expected that svd2rust will also grow AVR support -- Atmel has its own description file format: atdf -- using the atdf2svd approach. And there are XSVD (JSON version of SVD files) files for RISCV -- I don't think they are official though? -- so svd2rust could also grow support for RISCV via a xsvd2svd tool.

I have already written a library to parse ATDF files into data structures, which should be fairly easy to plug in.

It is [almost being] used in the ruduino library to implement a trait-based AVR-specific HAL. It is being used in another library that defines all registers/IOs for every AVR device.

dylanmckay commented 6 years ago

I like the idea of moving components into rust-embedded (and increasing the bus factor with that org 😂).

Agree strongly. I have mixed feelings on whether this should be done for AVR before it actually gets merged into upstream Rust though.

I've tried to be quite careful to manage expectations with the AVR fork - there's been quite a bit of interest, but the experience is still quite buggy. I think that moving some of the AVR stuff to the rust-embedded org might give off the impression that the AVR experience is well supported, where in fact things such as having optimisations disabled on a nontrivial project make it quite likely to have an LLVM assertion error.

I think there's a lot of value in bringing the embedded projects into one organisation. It'd definitely help the embedded experience become a lot more seamless, especially with strong collaboration on things such as HALs.

hannobraun commented 6 years ago

@japaric I'm happy to help out here and there with the cortex-m crates, but I don't have the bandwidth to drive any major initiatives.

Fixing this requires writing or putting together an SVD file that contains only core peripherals, running svd2rust over it and committing the code to the cortex-m repo.

I did some digging. I already knew about these incomplete SVD files, so at least there must have been some interest by ARM at some point.

I found this issue, specifically this comment, which states that core SVD files exist now, referring to some Keil pack. I found this download page, and I believe the pack that the comment refers to is "ARM V2M-MPS2 Board Support PACK for Cortex-M System Design Kit" under the "Keil" heading.

I downloaded the file (despite the name it's just a zip file; add ".zip" to the name and you should be able to open it), and yes, it contains SVD files for all the Cortex-Ms. I don't know if they're any good, and I don't know what the EULA says about using them, but I guess at the very least it makes sense to check them out.

Regarding organization I think it may make sense to move these core components under the rust-embedded org and then create, say, a cortex-m team in charge of maintaining the cortex-m crates.

:+1:

docbrown commented 6 years ago

@hannobraun It looks like the SVDs in that pack are for some kind of abstract Cortex-M device. They include peripherals (timer, GPIO, serial engine, etc) that aren't common to all Cortex-M designs. Maybe they're just copied directly from the Cortex-M System Design Kit?

@japaric If a SVD needs to be written from scratch, I can try to put one together directly from the Technical Reference Manuals so that there's no problem with licensing. Unless someone is already doing that.

hannobraun commented 6 years ago

@docbrown Ah, too bad. And I'm guessing we're not allowed to modify it (though I haven't read the EULA).

dvc94ch commented 6 years ago

So moved all riscv crates to a rust-riscv org. Should help find riscv crates, contributing etc. If someone wants to collaborate I can add you to the org. There is also a riscv-crates repo that creates a cargo workspace to make hacking on riscv crates easier - rust and llvm forks are updated. Some of these crates could be moved to rust-embedded once they're more mature.

docbrown commented 6 years ago

@hannobraun Yeah, it's kind of a minefield. There's the EULA (which appears to only reference example code and drivers, but IANAL), a 3-clause BSD(?) license from ARM on the SVDs themselves, and then another license from ARM applied to the code generated from the SVD (the licenseText element). And if the SVDs were just copied from the reference design in the Cortex-M System Design Kit, then there's likely a EULA for that, too (which we can assume Keil is in compliance with, but the kit doesn't appear to be publicly available to verify that). Writing an SVD from scratch seems like the safest option, IMO.

hannobraun commented 6 years ago

@docbrown

Writing an SVD from scratch seems like the safest option, IMO.

I'd try to contact ARM first, to try and clarify the license. That might not come to much, but it probably won't hurt either. But if you're willing to write an SVD from scratch, I'm not trying to stop you! Just offering suggestions.

Emilgardis commented 6 years ago

I'd be happy to help maintaining svd2rust. I'm also glad that were taking this direction because the pressure and responsibility has been something I've noticed to be far to big for a one-man team.

japaric commented 6 years ago

@Emilgardis @ryankurte

Thanks for volunteering!

I have just sent you collaborator invites for both japaric/svd and japaric/svd2rust. Those two are in a worse state than the cortex-m repos and I don't want to block development on those until we figure out an org structure.

@ryankurte

I'd like to see a code of conduct (..) as well as contribution guidelines (..) at the org level.

Great idea! Any suggestion on how to make it visible at the org level? I don't think you can have pinned repos at the org level. Though it seems you can have some description text at https://github.com/$ORG

@dylanmckay

I have already written a library to parse ATDF files into data structures

Awesome! I know that @kunerd and ecstaticmorse_ (IRC nickname) are interested in svd2rust support for AVR and/or a atdf2svd tool. You should all synchronize efforts.

scowcron commented 6 years ago

@dylanmckay @japaric re: ATDF

atdf2rust really would be nice to have, even for micros in the ATSAM families (Cortex-M). Atmel's SVD files are non-compliant and don't play nicely with svd2rust. I'd love to be able to generate code for those without have to hack svd2rust to ignore problems in the SVDs.

The other significant pain point I had working with svd2rust was lack of register group support, but I see that there's progress on pulling in svd v0.6.0, so I won't harp too much on that.

kunerd commented 6 years ago

@dylanmckay @japaric: ATDF

@ecstatic-morse created this atdf parser repo, which also can parse .atdf files into svd2rust structs, if I remember correctly. From my point of view it would make sense to separate the code generation part from the description file parsing part, so that for example svd2rust and atdf2rust tools can be created by using the same code generation base, either as a lib or by adding different executables to svd2rust. I think I can offer some time to support on this.

@dylanmckay: btw - thanks for the great work on the avr-rust fork

jamesmunns commented 6 years ago

@japaric I would be interested in also helping to maintain at least svd2rust. I have now spend 1/2 a week with it (due to the cluster issue), and feel more comfortable with it (but I am still far from being an expert).

japaric commented 6 years ago

@dylanmckay @dvc94ch Since you have org for core AVR / RISCV crates I would suggest you add a link to your orgs to the README of this repo. Also if you tag issues with the help-wanted label then when someone visits the org page they'll see how many help-wanted issues each repo under the org has.

@jamesmunns Awesome! Thanks. Sent you an invite.

@kunerd if the ATDF parser produces a svd2rust::Device struct then I think it would make sense to have an atdf2rust tool in the svd2rust repo. Could you open an issue in the svd2rust repo to discuss this?

ryankurte commented 6 years ago

@japaric org level code of conduct / contributor guidelines / how to contact are probably good candidates for rust-embedded.org? Then the org description can just be a link there and the main page already has a link back.

hannobraun commented 6 years ago

I don't think you can have pinned repos at the org level.

You can have pinned repositories for orgs! Might require some specific permission though, I don't know. See screenshot: org-pinned-repo

japaric commented 6 years ago

Sorry this has taken so long.

This my initial proposal for re-organizing the existing embedded repos under the rust-embedded org. Feedback is of course welcome!

Teams, collaborators and repos

Collaborators are split in two groups:

This is the tentative distribution of teams and repos. The initial membership reflects more or less things as they are right now.

The names after the tilde (~) indicate collaborators to that repo.

Code of conduct

All participants in rust-embedded projects will adhere to the Contributor Covenant Code of Conduct v1.4.

The code of conduct (CoC) will be linked from the rust-embedded org landing page as a pinned repository. A copy of the CoC will be included in each repository under the rust-embedded org (CODE-CONDUCT.md), and a note about the CoC will be included in the repository README (see failure for an example).

Teams and collaborators will enforce the code of conduct in the repositories they oversee and on the #rust-embedded IRC channel.


Thoughts?

Also, we need more people on the teams so this is your chance to volunteer to a specific team!

Finally, we need an e-mail address, or some other private channel, to receive reports related to violations of the CoC. @posborne @nastevens would be possible to set a mods@rust-embedded.org e-mail alias?

rahul-thakoor commented 6 years ago

Hello! I would like to help more. However, I am new to the Rust programming language and would definitely love to learn and contribute more, if I get assistance from this wonderful community! I am very interested in contributing to embedded-linux ecosystem.

Some thoughts:

  1. Are there plans to implement the serial interface for linux-embedded-hal? Could be interesting to interact with modules such as ESP8266 WiFi SoC or WISOL Sigfox module via UART.

  2. As of Linux 4.8 the new gpiochip char devices provides a new API and sysfs gpio is supposedly deprecated. There is already a C library, libgpiod, for interacting with the linux GPIO character device with C++ and Python bindings in the works. It would be great to have a Rust implementation.

  3. I can also help with French and Chinese(Simplified)() translation of awesome-embedded-rust or other resources if needed.

  4. Regarding maintaining the several projects, perhaps derek can help?

Thanks

therealprof commented 6 years ago

@japaric I'd like to get involved in the tools team and would like to add two important topics to the list which we should investigate: flashing and debugging, potentially also non-ITM logging (i.e. serial monitor support). I'd be definitely interested in lending a hand with cargo-binutils.

Potentially I'd also love to help with cortex-m though I currently have no idea what to contribute there.

ghost commented 6 years ago

Hello together. I'm also new to Rust but in software engineer business since 20 years now. Currently I'm not so happy with the organization of the different SVD files for the STM family of MCUs. @adamgreig is working on a promising approach to keep the SVDs from STM in one place and collect the needed patches there. Another point I'll like to help out are the currently spread implementations of the HAL driver implementations all across over different crates. Best, Ingo

musitdev commented 6 years ago

Hello, I can also give a hand and I've proposed to help in the refactoring of SVD. I develop with Rust since end 2012 and I have a few experience in embedded ARM Cortex (I work on a STM32f7 card for audio applications). I can help with french translation too.

posborne commented 6 years ago

As of Linux 4.8 the new gpiochip char devices provides a new API and sysfs gpio is supposedly deprecated. There is already a C library, libgpiod, for interacting with the linux GPIO character device with C++ and Python bindings in the works. It would be great to have a Rust implementation.

Hi @rahul-thakoor. I'm the author of the sysfs gpio crate. I have most of the core bits of a new crate based around the GPIO character device here: https://github.com/posborne/rust-gpio-cdev/tree/first-pass. I need to make some final decisions on APIs and document but an initial version should be on crates.io in a few weeks at the latest.

Once stabilized, this crate will likely move over to the rust-embedded org (which I created awhile back). I considered wrapping libgpiod but decided it made more sense to just interface with the kernel directly -- the API is very similar where it makes sense, but I have also strayed some in order to provide better safety/ergonomics using Rust's ownership system.

Both will need to be supported for some period of time, but I think it makes sense to start pushing people toward the cdev implementation if they know they will only be targeting 4.4+.

andre-richter commented 6 years ago

I would be happy to help on the resources team. Most probably I can contribute by reviewing material. Regarding resources, I am also in the process of porting tutorials for bare metal programming on RPi3 (aarch64, cortex-a) to Rust. Currently, the code is more or less literally translated from C, with some minor abstractions here and there. But it is definitely lacking a layered approach like there is with cortex-m, cortex-m-rt, cortex-m-quickstart, with all its safety abstractions etc. Therefore, I wonder if there are any plans for cortex-a in the not too distant future, or is that not on the radar?

berkus commented 6 years ago

Of these projects I'm probably able to help with embedded-hal somewhat, so will be happy to do it when i can, but alas won't take maintainership.

pftbest commented 6 years ago

Hello, @japaric! Can you please give me access to https://github.com/japaric/msp430-quickstart so I can merge https://github.com/japaric/msp430-quickstart/pull/3

rudihorn commented 6 years ago

I'd be happy to help manage PR's etc. with the https://github.com/japaric/stm32f103xx-hal crate.

korken89 commented 6 years ago

Hi @japaric, I have been working for a long time with the core Cortex-M parts for the C++ version of RTFM, where I am now (~slowly) transitioning to Rust. If you want help with the core cortex-m* crates, give me a ping.

japaric commented 6 years ago

I finally (!) wrote an RFC for this: #136. We'll discuss this RFC in the next embedded WG meeting (#130) which is in ~12 hours. If you can't make it to the IRC meeting but can comment on the RFC before the meeting please do! Even if everyone who attends the meeting is on board with the RFC we'll leave the RFC open for at least a week before we officially approve it. So leave comments over there if you have suggestions or concerns.