Combine RFM9X/RFM69 libraries

[jerryneedell]

Following up on https://github.com/adafruit/circuitpython/issues/7830 Is there any interest in creating a new library, say, Adafruit_RFM that combines the Adafruit RFM69 and RFM9X libraries since most of the code is (or could be) common to both. This would not be able to support the feather M0 RFM69 or RFM9X boards so we could leave the individual adafruit_rfm9x and rfm69 libraries as is. It may even be possible to add additional modules(NRF24L01 ?) This new library could incorporate new features and capabilities without concern for accommodating the M0 RFMxx boards.

There has also been discussion in #90 and #38 about creating a "headerless" protocol that does not append the RadioHead header. The would be compatible with other arduino and Python libraries that just send raw packets. It could be a new RFM_lite library or incorporated into the new library above.

I'd be happy to work on this or let someone else run with it if they are interested. Any thoughts?

I will post a link to this in the RFM69 library and ask that all discussion occur here.

[dhalbert]

Given that the RP2040 Feather RFM boards exist and cost less than the SAMD21 boards in addition, I think a more modern combined library that isn't constrained by the SAMD21 RAM size sounds like a nice idea, especially if you would like to work on it :) . You have a great deal of experience on this as well.

It could even have asyncio support, or there could be an asyncio version. I don't know if that's something people are asking for.

@ladyada ping for your opinion.

[ladyada]

hiya yeah the samd21 + RFM board predates circuitpy. a better library would be great!

[jerryneedell]

Great. I'll get to work on it. Should it be an Adafruit library or should I start it off as a Community Library?

[DJDevon3]

My understanding of the major differences between RFM 69 & 95 to keep in mind when consolidating. RFM69 has a much smaller message of about 32 bytes and RFM95 will do 255 bytes. You can send much longer messages with RFM95 to the point they can be used as P2P messengers.

The benefit of RFM69 is it's supposed to support basic encryption. I don't think encryption has been implemented in the RFM69 library yet so I would really like to see that feature added... due to an Amazon sidewalk interference (read below).

You're correct in that the libraries share a large amount of code and I think consolidating them is a good idea. Also even though RFM is only supposed to work in specific frequency ranges I've found I can set the frequency to whatever I want manually and it will still work... I think another feature that needs to be added is to only allow the library to work within a specific frequency range and throw an error if it's outside a legal range. I was surprised to find you can work around legal ranges too easily.

I'd like to see the library require a country code which automatically limits the range to stay within the respective country legal range. It will make it harder for people to accidentally or intentionally use the library outside of legal range.

I've used both libraries and boards for multiple projects, most notably my Mail Boombox project. I have examples using an RFM Featherwing with an ESP32-S2, M4 Express, and RP2040. https://github.com/DJDevon3/My_Circuit_Python_Projects/tree/main/Multi-Board%20Projects/LORA%20Mail%20Boombox That should help in seeing how most people would use the RFM boards.

I think the library should also set a node by default. One unfortunate side effect of not choosing a node is that Amazon Sidewalk will broadcast in a node-less fashion. It took a while to figure out why my RFM projects were being triggered when I'm pretty certain there's no one else in my area experimenting with RFM . It's because Amazon Sidewalk blasts messages quite regularly and will trigger anything not on a node scheme.

Anyone in range with an Amazon Sidewalk enabled device will trigger your RFM project out of the box.

Sorry for the wall of text, I worked with RFM extensively last year and ended up coming to the same conclusion there's very little difference in the libraries with about 95% of repeated library code. A variable switch to specify the board, country id, and other minor feature differences makes more sense.

As to the library name to amend 1 and deprecate the other, or deprecate both and create a new encompassing Adafruit_RFM type naming convention, I have no opinion. That's up to the devs to figure out what is most convenient for them as libraries, examples, and learn guides would be affected.

If you need help testing I can volunteer. I believe I have every different type of RFM board that Adafruit produced to date. Feel free to ping me if/when you're looking for beta testers.

[DJDevon3]

Here's one example of an RFM message I captured from what I suspect to be an Amazon Sidewalk device that was triggering my Mail Boombox randomly about every 5-10 minutes. If I understand how Sidewalk works it's a backup protocol to their WiFi enabled devices. If it cannot contact a Echo Dot, Echo Show, or other Sidewalk enabled device it will send an RFM message. I suspect it's just a basic ping of some kind. I was unable to decipher the message.

bytearray(b'C\x02U\xe2OD[\x07Cv\x83Q\xf7FA\xf7\xd5\xbf\xc0\xcawW\x0e@l\x18o\xd2{TR\x9d\xa0UB\x8e\x80\x8a\x9c\xa9@\xa9Pd;\x8a\xaao%\x9f\x82\x94J\xc7\xf3D\xd9\xb7\xb7\x01\x04\xa3\x0c\xb0\xa9\xe9\xd2\xb1\x0bJ)\xcc\xc72Gu\x1a\xc1\xd4\x95\xf3t\xc9\xbc\xa9\x88\xa7G\xa3WY\xc1+\xb8\x9a<D\x9c\x86f\x06\xe4\xcb\xb0\x18\x81\xf0\xbf\xf4pb\xdf\x07+A\xc1\xaa\x90\x84\xc8\xee\xb6!\x10\xc9Yav\xf1\xebFV\x05\xb0\xe1n\xb8\xbf\xf9T\x9e\xf5\x05\x11\xcc\x7f:-\xf2J\xc6\xca\xf1\x9fr$\xc1\xcfU6\xce\xc6Wh\x83\x1e\x18\xfc"T-\xf3bm\xc9\x84\xd5\x11\xb4\xfdV\xadJ\x8e}\xc0\xd7\x823\xe4\xe7#\xebhKy\\S\xd8\xd9\xbcA\xc0\x11\xdf\x89\xd7\xfd\xae)XQ\\\xd8\x91?\xed\xb7b\'\xd6\xff\xaa\xae\x9e\xc8\xeed\xb2\x91\xe63\xc8\xab\x16V*h=\x91\x00^@\xdc\x143\xfbe\x95')

I would get these types of messages hundreds of times per day until I switched to a node configuration. If anyone can figure out what it says I would love to know. I'm very curious about what type of data Sidewalk is casting via RFM. This is the type of interference you can expect with Sidewalk devices around.

[jerryneedell]

@DJDevon3 Thanks for your comments and suggestions.

Encryption is implemented. see the example https://github.com/adafruit/Adafruit_CircuitPython_RFM69/blob/main/examples/rfm69_simpletest.py#L38

The RFM69 packet length is up to to 60 bytes as implemented.

Theoretically , if encryption is disabled, the packet length can be up to 255 bytes, but that requires some much more complex data transfer to the internal FIFO and it is not implemented. I am not sure if encryption will help with your Amazon packet interference. You will still receive the packets, they will just be "unencrypted" garbage. According to an old issue, the CRC is enabled by default https://github.com/adafruit/Adafruit_CircuitPython_RFM69/issues/38 I'm not sure why the Amazon Sidewalk packets are getting through. As it is is implemented now, if the CRC fails, the packet is ignored. "Like it never even happened". It may need some additional code to implement a way to report the CRC errors as it has been on the RFM9x, I plan to look into that in this update. That may help with the "noise packet" issue you are having. Using the "node" address does help filter out random packets that are not wanted and is probably best for your "mailbox". I'm not sure it should be the default, but that can be discussed. It is certainly something to be aware of.

I am not convinced we want to implement the frequency limitations. As was noted on Discord, there are many things that impact the allowed use of a frequency. I'd prefer to leave that to the user, but I'll be interested to hear what other may think.

[jerryneedell]

@DJDevon3 Some additional information about Amazon Sidewalk. According to this https://docs.sidewalk.amazon/assets/pdf/Amazon_Sidewalk_Specification-1.0-rev-A-032823.pdf it uses LoRa which should not be an issue for the RFM69. It may explain some odd noise I see on me RFM9x units ;-) Why do you think you are seeing Amazon Sidewalk? If you can capture more examples, please post them.

[jerryneedell]

@DJDevon3 I stand corrected, enabling encryption does filter out unencrypted packets. It should help with avoiding your unwanted packets. Note, you have to set the key the same on both ends.

[jerryneedell]

@DJDevon3 Ahh -- Now I see that your Mailbox uses LoRa - RFM9x so it makes more sense that you would see interference from the Amazon Sidewalk... The CRC may not help because Amazon also may be enabling the CRC so the packets are indeed valid packets. I thin the Node addressing is the bast way to handle it in that case. I may have to get a "sidewalk" capable device to explore this further.(I'll have one by the end of the week:-)) It "should" not be an issue for the RFM69.

[DJDevon3]

I have many RFM69 boards but have yet to actually use them. RFM69 or RFM95 would work for my purposes on the mailbox project. I didn't know how many bytes I'd actually need. Turns out I only really need a few.

I suspect it's my Amazon Echo Show's specifically that are doing it. The correlation is walking into room which has both my mailbox and an Echo Show 5. The Echo Show has a PIR motion sensor built in which wakes it up. At the same time it wakes up, my mailbox goes off. It happened so many times that I couldn't chalk it up to coincidence.

I haven't touched my mailbox project in a while. Had to make room on my workbench for a huge 12 matrix panel project and the mailbox ended up on a shelf unplugged. I can plug it back in and capture more random packets no problem. It happens so consistently that grabbing more data for you won't be a problem. After thinking about it for a bit I'm not sure sharing the captured bytes was a smart idea. It might have personal information in it, I have no idea what's in the packet because I can't figure out how to decipher it.

It's like a weird form of encrypted hex as you can see from the bytearray. It might be gibberish but it's consistently gibberish in the same style so I suspect it's a custom encryption because it's an Amazon product. It would have had to pass FCC certification here, maybe there's FCC info on it. Since I haven't used RFM69 encryption yet and it sounds like you have do you see any correlation in the encryption scheme?

I cannot discount that it might just be RF noise from some other device. Setting the mailbox to a node did make the random triggers cease. Whatever is causing it is node-less. If it does turn out that Amazon is using node-less RFM blasting noise into the frequency I don't see how that would pass FCC certification.

Also I made an RFM Portable Rangefinder and the result was about 100 yards in a suburban environment. It doesn't have back scatter capability. So whatever is causing the interference is very very close at the least. Thanks to DanH for the project suggestion. It helped me get a feeling for my local range. Because I only used 2 transceivers at a time I knew exactly when a message was sent and received. The rogue messages that I was picking up were definitely not coming from my setup.

[jerryneedell]

The RFM69 should not even see the Amazon Sidewalk packets since they are LoRa and he RFM69 is using FSK. In addition, the encryption will help. But the range for the RFM69 is much less than for the RFM9X. I did notice in your Mailbox code, you are turning off the CRC. It may oar may not help to turn it on since the Sidewalk may well also enable the CRC. As you discovered, setting a node address also works well. The RFM9X and RFM69 libraries use a packet header that is compatible with the Arduino Radiohead library. That is what handles the node addressing and also allows for the "reliable datagram" mode where nodes can automatically exchange ACK packets to confirm receipt of a packet. Amazon Sidewalk is not using the RadioHead packet format but that does not stop them from being received by the RFM9X. I have an Echo arriving today so I hope to be able to test for interference.

I almost have a functional "combined RFM69/9X" library working. Lots of testing to do before I post it. So far, it does not add any features, just hopefully does not break anything....

One of my next projects is to try sending FSK packets from an RFM9X. The library now only uses LoRa. I'd like to see if it is possible for an RFM9X to actually communicated with an RFM69. If both use FSK, I am hoping they can.

[DJDevon3]

I spent most of yesterday attempting to break the encryption. I'm pretty sure the bytearray is encoded with some form of ASCII to Hex to SHA format. Just a matter of getting the right sequence. As long as there's no salt involved I might be able to reverse engineer it. The more we can learn about possible interference out there the better we can try to set some defaults to avoid it.

[DJDevon3]

After some research I found a rabbithole I'd rather not waste my limited brain power on.

A brief technical overview of Sidewalk encryption

The Sidewalk Network Server changes transmission IDs (TX-IDs) every 15 minutes to help prevent the tracking of devices and associating a device to a specific user. This packet will instruct a valid device to change its Network layer algorithm from AES-GCM to AES-CTR + AES-CMAC. The GW will continue dropping packets unless the authentication passes. This mode is valid until the next TX-ID period.

Every 15 minutes makes sense. What I thought was random was actually a pattern because I have 6 of these devices in my house. That's about 24 (seemingly random but not actually random) triggers in a 1 hour period or about 576 per day. That does seem to lineup with my experience, unfortunately. As you can imagine, it was driving me nuts before I found a way around it.

There's no way I'll break that encryption. Part of the salt and AES key are device ID, MAC, and time based. Unfortunately there is no reliable preamble to filter it out, cannot setup a blacklist to filter out Sidewalk devices. Instead we'll need to support a whitelist scheme where Adafruit RFM devices broadcast a preamble + encrypted message are only allowed. If the preamble is short with 2-3 bytes like "ADA" it wouldn't take much away from RFM69 bytes.

Yes we've already found a reliable way to avoid it but I think it's best to document this behavior because it will confuse the heck out of someone new to RFM that happens to own any Amazon Sidewalk enabled device in their home.

[jerryneedell]

I now have an Echo (4thgen) that is sidewalk enabled. So far, I have not seen any packets from it. I need to set up a good monitoring system. Are the packets you are seeing all the same size? What RSSI are you seeing? That is, are they strong signals? If you don't mind my asking,Do you also have a Ring doorbell? It sounds like they rely heavily on Sidewalk. I do not but I wonder if some very weak packets I receive may be coming from a neighbors doorbell.

Edited to add: I am also running an SDR and I do see activity around 915 -- I have several LoRaWan devices running, but as of yet I am not able to reproduce the RFM95 capturing anything I would attribute to the Echo and it does not see the LoRaWAn packets.

[jerryneedell]

WooHoo! I have finally succeeded in configuring an RFM9X for FSK and was able to send and receive between an RFM9X and an RFM69. Lots of clean-up and testing to do, but this was just to verify that it can be done! There are some limitations. The RFM9X does not support the AES encryption like the RFM69 and the maximum packet lengths are different, but I hope it will be useful to be able have these 2 modules be able to communicate with each other.

[DJDevon3]

@jerryneedell That's cool. Being able to merge them into 1 library would be really nice. It could host all the different examples in 1 repo too.

Yes I do have a Ring doorbell. It's an Amazon product too and does interface with the Echo Shows. The packet length is not consistent. That would make it easier. I can't specify the exact sizes from memory. They're not short messages. The example I posted seems about right but I do remember they seemed random in length.

[jerryneedell]

@DJDevon3 Ah -- I wonder if it is the Ring Doorbell, rather than the Echo that is the culprit.... So far, I cannot get my Echo to produce anything I can pick up with the RFM95

[jerryneedell]

Here is a link to my working repository https://github.com/jerryneedell/CircuitPython_RFM It is still quite early and "pre-alpha" but you are welcome to try it out and comment. It may change drastically at any point....

[jerryneedell]

OOK modulation also works for sending data between 2 RFM9X or between and RFM9X and an RFM69.

[DJDevon3]

@jerryneedell I have too many projects. :( No idea when I'll be able to get to this.

[jerryneedell]

@jerryneedell I have too many projects. :( No idea when I'll be able to get to this.

No worries... I was not expecting anyone in particular to test it. I have no idea when I'll be ready to release it. I'm still not convinced it is even worth doing this way. It was fun to add FSK/OOK to the RFM9x and be able to talk to an RFM69. This is the only really new feature so far. If anyone has a need for that or wants to test it, please give it a try.

[DJDevon3]

@jerryneedell I think combining the separate libraries makes a great deal of sense with the amount of repeated code. Don't doubt yourself, this is definitely the right direction. You even got them to communicate with each other. I think the concept of what you're doing is fantastic. It also ensures syntax will remain consistent. No one will have to bounce back and forth between 2 libraries for updates.

With the amount of repeated code and some refactoring I wouldn't be surprised if you actually manage to shrink the combined library to be smaller than either of the separate libraries.

[jerryneedell]

new library https://github.com/adafruit/Adafruit_CircuitPython_RFM.git has been created. The initial code has been submitted as a PR https://github.com/adafruit/Adafruit_CircuitPython_RFM/pull/1