ZZ-Cat / CRSFforArduino

An Arduino Library for communicating with ExpressLRS and TBS Crossfire receivers.
GNU Affero General Public License v3.0
143 stars 24 forks source link

feat(library): Serial Transmitter Interface #103

Open ZZ-Cat opened 6 months ago

ZZ-Cat commented 6 months ago

Overview

This Pull Request will bring in an exciting new feature: The Serial Transmitter Interface.

[!NOTE] This Pull Request focuses on bi-directional full-duplex non-inverted communication with the internal transmitter module in your handset.
The external module bay of your handset manages communication in a way that is outside the scope of this Pull Request, and it MAY warrant a Pull Request entirely of its own.

Details

If you recall the Serial Receiver Interface is the middleware layer between your Sketches (via the Sketch Layer) and your connected ExpressLRS or TBS Crossfire reciever (via the Hardware Abstraction Layer, which is managed by the Arduino API for back-end functionality).
The Serial Receiver Interface is so-called this because it implies CRSF for Arduino communicated with your connected ExpressLRS receiver. EG A RadioMaster RP3.

The Serial Transmitter Interface will add bi-directional communication to-and-from your handset's internal transmitter module.

Up until now, I have been focusing on the receiver-to-"flight controller" end of The Crossfire Protocol, and I had not even thought about the handset's side of the protocol... until it was brought to my attention by a group called Atopile and their Swoop project.
Atopile have asked me to put the Serial Transmitter Interface together, so that it MAY serve as the communications between the handset itself and its internal ExpressLRS transmitter module.


How this has impacted my quality-of-life and what can be done about it

This is the first time a company has approached me to implement a feature they need

In fact, Atopile are the first company I have ever worked with in this context, period.

At first, I thought this was pretty exciting and wanted to get it done ASAP for them.
However, as time marched on, the novelty of that wore off and reality set in... and that reality is "Oh f*ck. A bona fide company has asked me to essentially 'work for them' for free." Thus, putting me into the same bucket as most other FOSS developers out there.

Since that realisation, I have felt an unnecessary amount of pressure to get this done... even more than what I normally get when someone asks me to implement something. To be clear here, I can easily handle feature requests and what-not with no problems at all. Also, the company themselves have not done anything to make me feel this way.
It is simply me getting in my own head and over-thinking into Oblivion about it.

With that being said, it has still burned me out all the same.
Because of that, I have been stuck on how to actually go about this, and I had overwhelmed myself with "option paralysis" when I learned that on the handset side of things the internal and external transmitter modules handle things in completely different ways as far as their physical layers are concerned.
So, I figured it would be a good idea at the time to implement both in this Pull Request... only to burn myself out on option paralysis even harder.

So now, I need to remember to....

Focus on one feature at a time

In other words: Don't sacrifice myself in favour of my own project!

Instead of me being ambitious at the expense of my own quality-of-life, I have decided to drop implementing the external module bay version of the Serial Transmitter Interface for the time being.
I had a long time to think this one through, and the more I thought about it, the more I felt tackling both versions at once in one Pull Request was too overwhelming for me.

Atopile have only requested I implement communication to-and-from an internal transmitter module, and me doing both the internal transmitter module and external transmitter module bay communications would be me going above-and-beyond.... and what has that done to me as a developer? It burned me out to the point of getting dangerously close to shutting this project down.... which is something that I had a history of doing with other projects in my past. CRSF for Arduino is the only project of mine that has seen any success at all (for real: 100+ stars and 4,000 users is nothing to sneeze at), and that is what's currently keeping me from hitting that "delete repository" button. That success is all from you guys, and I am very grateful for that.

Tackling the much wider issues - Developer burnout, social engineering, and supply chain attacks

Currently, there is a lot on my plate as far as the upcoming Version 1.1.0 is concerned... probably a lot more than what I am used to. Most of which is all security-related stuff in light of the XZ Utils Backdoor incident.
I know, I know. I keep banging on about XZ Utils, and I probably should shut up about it. However, it's not just the back door itself. It's how it happened. It was a supply chain attack. A burned out sole maintainer was about to chuck in the towel on their project, and some clever cybercriminal named Jia Tan came along and socially engineered the maintainer into making them (Jia Tan) the primary maintainer before they (the original maintainer) left the project.
Over time, Jia Tan got to work on writing (and implementing) dodgy code... and a few years later... hey presto, we have a back door on our hands what's slated to be released into the wild that could have spelled devastating results to an ecosystem what prides itself on its security.

I keep bringing up XZ Utils, 'cause I be like "Who's to say the same thing won't happen to me?" 'Cause I am in the exact same position - A burned out and overwhelmed solo maintainer of (an increasingly) widely used project.
Okay, sure, I write firmware, and CRSF for Arduino is a firmware library that targets microcontrollers. How could malware possibly be deployed to that? Quite easily. In fact, it's easier than you may think. All a threat actor would need to do is convince me that I can trust them (good luck with that, 'cause I got severe trust issues to begin with, and because I'm neurodiverse, social engineering doesn't work on me anyway), they gain access to my project, do their thing, and the next thing you know... malware has been covertly deployed under the guise of my legitimate project... and I could be powerless to do anything about it, because either I have been removed from the project (because the cybercriminal was granted admin priveleges) or I decided "enough is enough" with my project and door-slammed and completely washed my hands of it. That is a situation I don't want to happen to my project, and I am taking all steps I deem necessary to negate the chances of that happening, at all.

This begs the next question: Who would want to inject malware into my project in particular? Well, ask yourself this: Who would want to come up to you while you're on your daily commute to/from work, and mug you for your money and your ID, and impersonate you while they get up to the most heinous of crimes? If you answered "that will never happen to me", then that's all the more reason why I am doing this. Because "it will never happen to me" is the epitome of complacency. It is that complacency that I view as being equal parts as dangerous as that identity thief in my analogy. I feel as a responsible maintainer, I have an obligation to protect my project from supply chain attacks and any other potential vector a cybercriminal could use to hijack CRSF for Arduino.

Coming full circle - Reducing burnout

I have decided to return to my "old schedule" where I work only between two and four hours per working day, and that is only in the mornings, during the week and on days where I'm not in physical pain (I also have rheumatoid arthritis, which can severely impact my ability to do literally anything, let alone code).

At this point, I MUST prioritise taking care of myself before I can take care of my project(s).
Because when I am constantly burned out, how can anyone expect me to write any code at all, let alone anything that's of good quality?
Up until a few weeks ago, my philosophy with CRSF for Arduino has been "Quality over quantity" and "less is more".
Why did I change this? Because I let my own intrusive thoughts win out, and I fell back into old habits what led me to the current state I am in. So, yes, burning myself out is my own fault, and this is what I am doing to rectify that.

What this means to you

This means that it will take longer for things to be implemented, yes. So, please be patient.
I would rather provide you all with a code-base that is of good quality that does what it says on the tin, as opposed to a code-base that is focused on sacrificing quality in favour of quantity.

...and after seeing several great projects become abandonware, I prefer to not abandon CRSF for Arduino. Especially when I am well aware of how popular it is becoming. You guys are what's holding me accountable here, and I love it. =^/.~=

ZZ-Cat commented 5 months ago

It appears I may have hit a wall with this.
According to the documentation, external CRSF transmitter modules are using inverted half-duplex UART.
This could severely limit my options here, because a lot of supported targets don't have the ability to invert their GPIO pins what service UART. The SAMD51J19A being one of those.

I'm not entirely sure on where to go from here, because of that actually. I really feel like I have hit a wall here.

Probably the best thing for this Pull Request for the time being is to have it so that the Serial Transmitter Interface only works with internal transmitter modules, because those use full-duplex non-inverted UART. That is more straightforward.

Writing full-duplex is incredibly straightforward, because it means I don't need to get into the bare metal side of things. Whereas with half-duplex, that's a little more involved, and not all hardware natively supports it (let alone the ability to assign custom UART pins). Even with targets that support it, I still need to write the parts where one pin and pad combination is swapped over (EG from Tx to Rx and vice versa), and that almost invariably involves bare metal coding - IE Dicking around under the hood with direct hardware registers.

IIRC, I did enable the ability to assign custom UART pins on supported hardware... so, it's likely that the Serial Transmitter Interface may only work with these targets if someone wants to use CRSF for Arduino with an external module (such as the RadioMaster Ranger).

ABLomas commented 5 months ago

One transistor and resistor can solve inversion problem, so maybe this is not big deal?

Also, would like to clarify - i would be interested if it can just "output uninverted signal", just for TX, something like: image (in other words - read channel values, modify them and output to FC) You may ask "is there anything that FC cannot do?" and i would answer "yes". For example - many GPIO's (on pi side) and autopilot mode trigger based on inputs, stepper motor control (using the same control channel, for example ch1-8 for FC, 9-16 for Pi with other stuff) and so on, so on. This would not require inverted output, just two separate UART's, right?

ZZ-Cat commented 5 months ago

One transistor and resistor can solve inversion problem, so maybe this is not big deal?

Also, would like to clarify - i would be interested if it can just "output uninverted signal", just for TX, something like: image (in other words - read channel values, modify them and output to FC) You may ask "is there anything that FC cannot do?" and i would answer "yes". For example - many GPIO's (on pi side) and autopilot mode trigger based on inputs, stepper motor control (using the same control channel, for example ch1-8 for FC, 9-16 for Pi with other stuff) and so on, so on. This would not require inverted output, just two separate UART's, right?

GET OUTTA MY HEAD!!! 🤣

No, seriously. This is why I spit ball my ideas in the comments section of my Pull Requests like this. Because I know someone can chime in and give me a hand on something that I could be having a hard time with on my own; and that's what this project is all about.

...and what you have suggested here has given me some extra food for thought.

I was thinking of starting the Serial Transmitter Interface off with standard non-inverted full duplex (IE separate Tx and Rx lines), because this is more straightforward to implement. Then possibly adding half-duplex on targets that are able to support it (through customisable UART pin assignments), and having the option to have it inverted on (currently few and far between, according to my information) targets that support GPIO inversion.

As far as I am aware, internal transmitter modules (such as the one in the RadioMaster TX16S Mk2) use standard full-duplex non-inverted UART, and the Rx and Tx lines are separate (like what we're all familiar with, on our receivers).


in other words - read channel values, modify them and output to FC

This would not require inverted output, just two separate UART's, right?

That's in the context of receiver-to-flight-controller.
Therefore your physical layer is two separate Tx and Rx lines, full-duplex non-inverted UART.
So, no. Hardware inversion isn't needed there.

I would be interested if it can just "output uninverted signal"

Yup. Can be done. As I said, that's straightforward to implement.

ZZ-Cat commented 5 months ago

Right! Now, I know where to go with this.

I am focusing on getting a "working prototype" of the Serial Transmitter Interface, where I am sending out the CRSF RC Channels packet at an arbitrary packet rate over non-inverted full-duplex UART.

I have created a file where development on this is taking place. It currently prints the phrase "Hello, World!" to the terminal, simply because I wanted to get the development environment set-up and ready-to-go.

The environment itself is slightly different from how I used to do it, in the fact that I have turned on several build flags related to compile-hardening. I have done this, so I can practically shoot two stones with one bird, because I want to bring in compile hardening to CRSF for Arduino's build process... especially in the context of my Quality Control.
So, now is a good time to trial-run that at the same time as I am working on the Serial Transmitter Interface.

Additionally, this development environment only builds the target that I am currently using to develop the Serial Transmitter Interface. Currently, this target is my tried-and-true Adafruit Metro M4 Express.
Eventually, I will port over to one of the RP2040 targets, because I have a fair amount of folks with RP2040-based targets watching this development with great anticipation.

Once development on this is complete, I will disable the development environment (in the same way as I have always done) and re-enable the entire build process across all compatible targets.

ZZ-Cat commented 5 months ago

Progress update 2024-4-26

I now have a serial_transmitter_prototype.cpp in the $PROJ_DIR/examples/platformio directory, which was introduced in 86e9337cbfb34aeebdca5894949e701a9886824b.
As of 0515624a096efcb15cd2960ab5645f30be04eaa1, a blank full CRSF packet is sent out on Serial1 at a packet rate of 50 Hz. This test lasts for about three seconds and terminates.

The prototype itself watches the micros() counter during execution, and derives a time->time_us_error from the selected packet rate (in this case, the selected packet rate is 50 Hz, but all of ExpressLRS' standard packet rates (as of ExpressLRS v3.3.0) are available). This error is compared against a maximum allowed error time->time_us_max_allowed_error that's set at 2 µS.
When time->time_us_max_allowed_error is exceeded, the test terminates with an error showing how far out the time->time_us_error is.

I am still "fleshing out" the CRSF RC Channels packet itself.
Once I have that done, the serial_transmitter_prototype.cpp will send out centred (except for ch5, which will be locked to 1000 µS) RC channels, along with the correct CRC.

Effectively creating a "proof-of-concept" that I can refine as time marches on.

ZZ-Cat commented 4 months ago

Right-oh. Changing tack with this slightly.... well... more accurately, keeping it in-line with the rest of the library where I am not providing any drivers at all. You write/bring your own.
By the time I have encapsulated this into the code-base, you will be REQUIRED to create your own means of scheduling of sending out and receiving CRSF packets.

Doing it this way offloads the need for me to maintain my own drivers on top of the rest of CRSF for Arduino. This is precisely the reason why I made it so from the very beginning that this library does not provide its own drivers for anything. You essentially provide your own, and adapt CRSF for Arduino to that instead.

From what I understand of the implementations of the handset's side of the protocol is... internal transmitter modules use the same hardware layer as what we are using on our receiver-to-flight-controller side of things: Separate Rx and Tx lines that are full duplex and are non-inverted.
The external transmitter module bay is more complicated in the sense that it relies on a combined Rx/Tx line that is half-duplex and the data signal is inverted. This complicates things to the extent where it is outside the scope of this Pull Request, as this Pull Request dominantly focuses on communicating with an internal transmitter module.

I will update the OP shortly, to reflect this.

dbloemhard commented 3 months ago

Found this project while looking for ELRS libraries for a small transmitter project I am doing.

The Inverted UART should be a non-issue. Users can select whether CRSF is inverted or not on the TX module side, at the time of flashing. So this just needs to be a caveat in your firmware - be sure to uncheck the inverted UART checkbox when flashing your TX module. Its the same requirement for DeviationTX transmitters, and i see the same note in the SimpleTX project.

ZZ-Cat commented 3 months ago

Found this project while looking for ELRS libraries for a small transmitter project I am doing.

The Inverted UART should be a non-issue. Users can select whether CRSF is inverted or not on the TX module side, at the time of flashing. So this just needs to be a caveat in your firmware - be sure to uncheck the inverted UART checkbox when flashing your TX module. Its the same requirement for DeviationTX transmitters, and i see the same note in the SimpleTX project.

Thank you for the tip. =^/.^=

I have had MAJOR Writer's Block over the last few weeks because of this.

dbloemhard commented 3 months ago

Definitely wouldn't stress over it :) All ELRS modules are expected to be flashed at some point, so you just add a point that CRSF for Arduino library needs the transmitter module to be flashed with the Inverted UART option unchecked and thats all. I am sure there are more difficult parts to code without worrying about signal inverters!

ZZ-Cat commented 3 months ago

Definitely wouldn't stress over it :) All ELRS modules are expected to be flashed at some point, so you just add a point that CRSF for Arduino library needs the transmitter module to be flashed with the Inverted UART option unchecked and thats all. I am sure there are more difficult parts to code without worrying about signal inverters!

Yeha.
The folks what originally spurred this into action simply wanted me to make it so my library communicates with the internal module of their controller.
That is easy for me to implement, because the protocol for internal modules is basically the same as what we have between our receivers and flight controllers: Separate Tx and Rx lines.
So, it's trivial for me to write the code. Therefore, this is what I am deciding on implementing first.

The communication between your controller and your external module is half-duplex and it relies on a single wire line. Some of the targets my library supports may require additional hardware in order to make that single wire line possible, if it is not possible to configure a chipset's internal UART and respective pins to service that.

At this stage, I'll tackle the internal transmitter module side of things first, and get this Pull Request squared away. Then, at a later date, I may look into the external module side of things and open a separate Pull Request for that, just to cut down on my perception of this being "HOLY SHIT, THIS IS SO OVERWHELMING!!!"

dbloemhard commented 3 months ago

Interesting. So I am using an RX flashed as a TX, with one of the receivers that has a 100mw PA onboard (100mw has been enough for me for all of the flying i do)... With the TX firmware flashed by default it becomes the same full duplex transmitter module with the RX and TX pads both being used. So this PR could be very useful to me as well.

Right now i am using the SimpleTX project as it is just a simple transmitter, no screen, no options to be set. But if your PR works, i may switch to using this library. Eventually the goal is to make a motion controller though, basically exactly what Swoop is, but no custom PCBs and as small as possible. I'd like to use a Betaflight FC flashed with the special firmware that Limon FPV described in his recent video (so it outputs gyro data via SBUS), an Arduino to translate that into CRSF, and then the RX flashed as a TX to then push that out to a tiny little whoop or micro drone.

ZZ-Cat commented 3 months ago

Interesting. So I am using an RX flashed as a TX, with one of the receivers that has a 100mw PA onboard (100mw has been enough for me for all of the flying i do)... With the TX firmware flashed by default it becomes the same full duplex transmitter module with the RX and TX pads both being used. So this PR could be very useful to me as well.

Right now i am using the SimpleTX project as it is just a simple transmitter, no screen, no options to be set. But if your PR works, i may switch to using this library. Eventually the goal is to make a motion controller though, basically exactly what Swoop is, but no custom PCBs and as small as possible. I'd like to use a Betaflight FC flashed with the special firmware that Limon FPV described in his recent video (so it outputs gyro data via SBUS), an Arduino to translate that into CRSF, and then the RX flashed as a TX to then push that out to a tiny little whoop or micro drone.

What's funny is... the guys that asked me to do this are the same folks that are working on Swoop. They sent me one of their prototypes to test this Pull Request on.
So, I can f*ck around and find out here. =^/.~=

Woz4tetra commented 2 months ago

hey, I'm interested in this project for my autonomous vehicle application. I'd be willing to help out with development.

I've got a TBS crossfire transmitter and a robot equipped a TBS crossfire rx nano. I see you have a test file with some basic communication implemented. Have you tested this on hardware? How do you wire up an Arduino to a crossfire transmitter? I read that crossfire is half duplex. I don't think Arduino supports this protocol out of the box.

Woz4tetra commented 2 months ago

ah I read your earlier message. I'll look into hardware that converts to full duplex. Do you have hardware you recommend to do this?

ZZ-Cat commented 2 months ago

I've got a TBS crossfire transmitter and a robot equipped a TBS crossfire rx nano. I see you have a test file with some basic communication implemented. Have you tested this on hardware?

I am yet to test the Serial Transmitter Interface in the real world. Currently, I am not at this stage yet.
Yes, I do have basic functionality, but that SHOULD NOT be relied on at this time, as that is very much my draft I created to give me an idea of what the guts-and-feathers of the Serial Transmitter Interface might look like.

How do you wire up an Arduino to a crossfire transmitter?

Currently, there is no support for external transmitter modules.

As far as internal transmitter modules are concerned, I am currently targeting Swoop's internal transmitter module and their RP2040 host microcontroller.
The makers of Swoop were the ones that requested this first, therefore support for their hardware takes priority here.
I am aware that they have followed the guidelines of CRSF Working Group for designing the physical layer of their internal transmitter module... which is full-duplex, and easy for me to work with. I am also following the same guidelines from the CRSF Working Group to ensure my library is compliant with their specifications of the Crossfire Protocol.

I'll look into hardware that converts to full duplex.

That may not be necessary, because of :point_down: this.

I read that crossfire is half duplex.

That's only half true.
The Crossfire Protocol, as stipulated by the CRSF Working Group, uses full-duplex UART between the receiver and flight controller, and between the handset and the internal transmitter module. half-duplex UART is used to communicate with an external transmitter module via the handset's S.PORT.

I don't think Arduino supports this protocol out of the box.

The existence of CRSF for Arduino proves that statement wrong.
If it didn't support it at all, I would not have written this library. I would have written something else.
To back that up, I have a Hardware Compatibility section in CRSF for Arduino's Wiki which features an exhaustive list of every Arduino-compatible development board CRSF for Arduino is compatible with.
If your particular board is not on that list, and it meets the minimum requirements you're welcome to request it to be added.

Do you have hardware you recommend to do this?

No. I don't do hardware recommendations. Instead, I encourage you to research what's available, do your due diligence, and make your own decisions based on the information you have available to you. I am not the type what tells you "buy this", "use that", "don't use x"... no... just... no.

I'd be willing to help out with development.

If you're serious about this, then you need to take the time to read through my Code of Conduct, Contribution Guidelines, and CFA's new license.

[!NOTE] You MUST have GPG or SSH commit signature verification enabled before your first commit to my code-base.
I have links at the bottom of my Contribution Guidelines which will take you to GitHub's instructions page on how to set this up. Yes, it can be a pain to set-up. However, it is a requirement for your Pull Request(s) to be taken into consideration. Pull Requests with one or more commits that have anything other than a green Verified badge will be rejected outright, as they simply cannot be merged with the development branch or the Main Trunk.

If you wish to contribute... There is now a development branch available for development, testing, and evaluation purposes for the upcoming Version 1.1.0 release.
You MAY fork CRSF for Arduino with this development branch intact, and base your branch off of the Version-1.1.0-Development branch. Then, submit your changes back here in the form of a Pull Request. Keep in mind that your Pull Request MUST be made to the Version-1.1.0-Development branch.

Keep in mind that simply because you submitted it, that does not guarantee your Pull Request will be merged, and I reserve the right to reject your Pull Request for any reason I deem necessary. Usually I will tell you why it got rejected in the first place (out of courtesy). However, I am under no obligation to tell you why I rejected it.

ZZ-Cat commented 1 month ago

Screenshot from 2024-09-01 11-20-10

=-/.-=
One tedious re-base down, and apparently I have one more to go.

ZZ-Cat commented 1 month ago

Finally!
Got it re-based successfully. =^/.^=