search

antirez / freakwan

A floor-routing WAN implementing a chat over bare-LoRa (no LoRaWAN).
BSD 2-Clause "Simplified" License
390 stars 22 forks source link

readme

FreakWAN

FreakWAN is an effort to create a LoRa-based open WAN network, completely independent from Internet and the cellular phones networks. The network built with FreakWAN has two main goals:

  1. To provide both a plaintext and an encrypted distributed chat system, that can be used by technology amateurs, or in places where internet is not available and during disasters.
  2. As a side effect of the first goal, to create a robust protocol over LoRa to support other applications, like sensors data collection, home automation applications, not having the usual range limitations of OOK/FSK communcation, and so forth.

Our goal is to cover parts of the Sicily with such a network. The code will be freely available to anyone wanting to build their own LoRa WANs on top of this software. The main features of our implementation and protocol are the following:

This code is currently a functional work in progress, designed to work with the following ESP32-based devices:

  1. LILYGO TTGO T3 v2 1.6 LoRa module.
  2. LILYGO TTGO T3-S3 v1.2 LoRa module (2.4 Ghz version not tested/supported).
  3. LILYGO TTGO T Beam LoRa module.
  4. LILYGO T-WATCH S3.

However changing the pins in the configuration, to adapt it to other ESP32 modules that have an SX1276, SX1262 LoRa chips, and an SSD1306 or ST7789 display (or no dislay, in headless mode), should be very little work. T-ECHO devices are also supported, even if with less features, in the C port of FreanWAN, under the techo-port directory, but the T-ECHO port is still alpha quality software.

FreakWAN is implemented in MicroPython, making use only of default libraries.

Installation

(NOTE: you need the : below, is not an error)

mpremote cp *.py :

or

talk32 /dev/tty.usbserial001 put *.py

Please note that you don't need both the command lines. Just one depending on the tool you use.

Usage

The two simplest ways to send commands to the device, write messages that will be broadcasted and also receive messages sent by other users, it is to use the USB or Bluetooth serial.

Serial CLI

To obtain a serial command line interface, make sure the device is connected via an USB cable with your computer. Than connect to the device serial with talk32, minicom, screen or whatever serial terminal you want to use. Normally the bound rate is 115200. Example of command lines and tools you could use:

mpremote repl

or

talk32 /dev/tty.usbserial001 repl

or

screen /dev/tty.usbserial001 115200

Of course the name of the device is just an example. Try ls /dev/tty* to see the list of possible device names in your computer.

Once you connect, you will see the device logs, but you will also be able to send bang commands or messages to the chat (see below).

Bluetooth low energy CLI

It is possible to use the device via Bluetooth LE, using one of the following applications:

Sending commands and messages

Using one of the above, you can talk with the device, and chat with other users around, sending CLI commands. If you just type some text, it will be sent as message in the network. Messages received from the network are also shown in the serial console. If you send a valid command starting with the ! character, it will be executed as a command, in order to show information, change the device configuration and so forth. For now you can use:

New bang commands are added constantly, so try !help to see what is available. We'll try to take this README in sync, especially after the first months of intense development will be finished.

Using the device via Telegram

When FreakWAN is located in some fixed location with WiFi access, it is possible to put it online as a Telegram bot. This way it is possible to receive the messages the device receives via LoRa as Telegram messages, and at the same time it is possible to send commands and messages writing to the bot.

To use this feature, follow the instructions below:

  1. Create your bot using the Telegram @BotFather.

  2. After obtaining your bot token (it's basically the bot API key) use the following commands in the FreakWAN cli (either via USB or BLE) or alternatively edit wan_config.py to set the same parameters.

    !wifi add networkname password !wifi start networkname !telegram token !telegram start

Now use your Telegram application in order to sent !help to the bot, and wait for the reply. If you can receive the reply correclty, Freakwan will also set the target of your messages, that is, the account you used to talk with the bot the first time. Now you are ready to save your configuration with:

!config save

Using the device via IRC

FreakWAN is able to join IRC, as a bot. It can receive messages and commands via IRC, and also show messages received via LoRa into an IRC channel. Edit wan_config.py and enable IRC by setting the enabled flag to True, and configuring a WiFi network. Upload the modified file inside the device and restart it. Another way to enable IRC is to use bang commands via Bluetooth, like that:

!wifi add networkname password
!wifi start networkname
!irc start
!config save (only if you want to persist this configuration)

The device, by default, will enter the ##Freakwan-<nickname> channel of irc.libera.chat (please, note the two # in the channel name), and will listen for commands there. The same commands you can send via Bluetooth are also available via IRC. Because of limitations with the ESP32 memory and the additional MicroPython memory usage, SSL is not available, so FreakWAN will connect to IRC via the TCP port 6667, which is not encrypted.

Encrypted messages

By default LoRa messages are sent in clear text and will reach every device that is part of the network, assuming it is configured with the same LoRa frequency, spreading, bandwidth and coding rate. However, it is possible to send encrypted messages that will reach only other users with a matching symmetric key. For instance, if Alice and Bob want to communicate in a private way, they will set the same key, let's say abcd123, in their devices. Bob will do:

!addkey alice abcd123

While Alice will do:

!addkey bob abcd123

(Note: they need to use much longer and hard to guess key! A good way to generate a key is to to combine a number of words and numbers together, or just generate a random 256 bit hex string with any available tool).

Now, Alice will be able to send messages to Bob, or the other way around, just typing:

#bob some message

Bob will see, in the OLED display of the device, and in the Android application, if connected via BTE, something like that:

#alice Alice> ... message ...

Encrypted messages that are correctly received and decoded are shown as:

#<keyname> Nick> message

Each device can have multiple keys. The device will try to decrypt each encrypted message with all the keys stored inside the key chain (the device flash memory, under the keys directory -- warning! keys are not encrypted on the device storage).

If many users will set the same key with the same name, they are effectively creating something like an IRC channel over LoRa, a chat where all such individuals can talk together.

This is the set of commands to work with encrypted messages:

#<keyname> some message     -- send message with the given key.
!addkey keyname actual-key-as-a-string  -- add the specified key.
!delkey keyname             -- remove the specified key.
!keys                       -- list available keys.
!usekey keyname             -- set the specified key as default to send all the next messages, without the need to resort to the #key syntax.
!nokey                      -- undo !usekey, return back to plaintext.

Sending images

FreakWAN implements its own very small, losslessy compressed 1 bit images, as a proof of concept that we can send small media types over LoRa. Images are very useful in order to make the protocol more robust when working with very long packets (that have a very large time on air). Inside the fci directory of this repository you will find the specification of the image format and run length compression used, and a tool to convert small PNG files (255x255 max) into FCI images. For now, only images that are up to 200 bytes compressed can be transmitted.

Once you have the FCI images, you should copy them into the images directory inside your device (again, using ampy or talk32 or any other tool). Then you can send the images using the !image command using the bluetooth CLI.

You can find a couple test FCI images under fci/testfci. They are all less than 200 bytes, so it is possible to send them as FreakWAN messages.

Power management

Right now, for the LILYGO TTGO T3 device, we support reading the battery level and shutting down the device when the battery is too low, since the battery could be damaged if it is discharged over a certain limit. To prevent such issues, when voltage is too low, the device will go in deep sleep mode, and will flash the led 3 times every 5 seconds. Then, once connected again to the charger, when the battery charges a bit, it will restart again.

For the T Beam, work is in progress to provide the same feature. For now it is better to disable the power management at all, by setting config['sleep_battery_perc'] to 0 in the wan_config.py file.

If you plan to power your device with a battery that is not 3.7v, probably it's better to disable this feature from the configuration, or the device may shut down because it is sensing a too low voltage, assuming the battery is low.

FreakWAN network specification

The rest of this document is useful for anybody wanting to understand the internals of FreakWAN. The kind of messages it sends, how messages are relayed in order to reach far nodes, the retransmission and acknowledge logic, and so forth.

The goals of the design is:

  1. Allow far nodes to communicate using intermediate nodes.
  2. To employ techniques to mitigate missed messages due to the fact LoRa is inherently half-duplex, so can't hear messages when transmitting.
  3. Do 1 and 2 considering the available data rate, which is very low.

Message formats

The low level (layer 2) format is the one with the explicit header selected, so it is up to the chip to add a length, a CRC and so forth. This layer is not covered here, as from the SX1276 / SX1262 driver we directly get the clean bytes received. So this covers layer 3, that is the messages format implemented by FreakWAN.

The first byte is the message type byte. The following message types are defined:

The second byte of messages of all the message types is the flag byte. Bits have the following meaning:

Currently not all the message types are implemented.

DATA message

Format:

+--------+---------+---------------+-------+-----------+------------------//
| type:8 | flags:8 | message ID:32 | TTL:8 | sender:48 | Message string:...
+--------+---------+---------------+-------+-----------+------------------//

Note that there is no message length, as it is implicitly encoded in the previous layer. The message string is in the following format:

+--------+---------------------//
| nlen:8 | nick+message text
+--------+---------------------//

Where nlen is a byte representing the nick name length inside the message, as an unsigned 8 bit integer, and the rest is just the nickname of the specified length concatenated with the actual message text, without any separator, like in the following example:

"\x04AnnaHey how are you?"

The TTL is set to 255 normally, and decreased at every retransmission. The sender ID is the HMAC returned by the device API, while the 32 bit message ID is generated randomly, and is used in order to mark a message as already processed, in order to discard duplicates (and there are many since the protocol uses broadcasted retransmissions in order to build the WAN).

Note that on retransmissions of the same message by other nodes, with the scope of reaaching the whole network, the message sender remains set to the same sender of the original message, that is, the device that created the message the first time. So there is no way to tell who sent a given retransmission of a given message.

Data messages may contain media in case this flag is set in the header of the message:

When this happens, the data inside the message is not some text in the form nick+message. Instead the first byte of the message is the media type ID, from 0 to 255. Right now only two media types are defined:

+--------+------//------+-----------+-------------+----//
| type:8 | other fields | sender:48 | mediatype:8 | ... media data ...
+--------+------//------+-----------+-------------+----//

Devices receiving this message should try to show the bitmap on the screen, if possible, or if the image is too big or they lack any graphical display ability, some text should be produced to make the user aware that the message contains an image.

ACK message

The ACK message is used to acknowledge the sender that some nearby device actually received the message sent. ACKs are sent only when receiving messages of type: DATA, and only if the Relayed flag is not set.

The goal of ACK messages are two:

  1. They inform the sender of the fact at least some near nodes (immediately connected hops) received the message. The sender can't know, just by ACKs, the total reach of the message, but it will know if the number of receivers is non-zero.
  2. Each sender takes a list of directly connected nodes, via the HELLO messages (see later in this document). When a sender transmits some data, it will resend it multiple times, in order to make delivery more likely. To save channel time, when a sender receives an ACK from all the known neighbor nodes, it must suppress further retransmissions of the message. In practice this often means that, out of 3 different transmission attempts, only one will be performed.

The reason why nodes don't acknowledge with ACKs messages that are relayed (and thus have the Relayed flag set) is the following:

Format:

+--------+---------+---------------+-----------------+---------------+
| type:8 | flags:8 | message ID:32 | 8 bits ack type | 48 bit sender |
+--------+---------+---------------+-----------------+---------------+

Where:

HELLO message

This message has the unique goal of advertising our presence to other devices in the network. This way, when a new device, part of the WAN, is powered on, it can tell if it is alone or surrounded by one or more other participants that are near enough to be received.

Hello messages are sent periodically, with a random period between 60000 and 120000 milliseconds (one to two minutes).

Devices receiving HELLO messages will compile a list of neighbors. A device is removed from the list if we don't receive a HELLO message from it for 10 minutes (this means we need to miss many successive hello messages, in order to remove a device -- this is an important point, since we need to account for the high probability of losing messages for being in TX mode while some other node broadcasts).

Format:

+--------+---------+---------------+--------+------------\\
| type:8 | flags:8 | 48 bit sender | seen:8 | status message
+--------+---------+---------------+--------+------------\\

Messages relay

Data messages with the PleaseRelay flag set are retransmitted by the nodes receiving them. The retransmission is the fundamental way in which the WAN is built. Imagine the following FreakWAN set of devices:

A <------ 10 km -------> B <----- 10km -----> C

For a message sent by A to reach C, if we imagine a range of, for instance, 12 km, When B receives the messages created by A it must repeat the messages, so that C can also receive them.

To do so, FreakWAN uses the following mechanism:

  1. A data message that has the PleaseRelay bit set, when received, is retransmitted multiple times, assuming its TTL is still greater than 1. The TTL of the message is decremented by one, the Relayed flag is set in the message, finally the message is sent again as it is, without changing the sender address, but maintaining the original one.
  2. Devices may chose to avoid retransmitting messages with a too high RSSI, in order to avoid using precious channel time without a good reason. It is of little interest that two very nearby devices retransmit their messages.
  3. Retransmitted messages have the Relayed flag set, so ACKs are not transmitted by the receivers of those messages. FreakWAN ACKs only serve to inform the originator of the message that some neighbor device received the message, but are not used in order to notify of the final destinations of the message, as this would require a lot of channel time and is quite useless. For direct messages between users, when they will be implemented, the acknowledge of reception can be created on top of the messaging system itself, sending an explicit reply.
  4. Each message received and never seen before is relayed N times, with N being a configuration inside the program defaulting to 3. However users may change it, depending on the network nodes density and other parameters.

Listen Before Talk

FreakWAN implementations are required to implement Listen Before Talk, in order to avoid sending messages if they detect some other valid LoRa transmission (either about FreakWAN or not) currently active. In this implementation, this feature is accomplished by reading the LoRa radio status register and avoiding transmitting if the set of bits reports an incoming packet being received.

LBT is a fundamental improvement for the performance of this protocol, since a major issue with this kind of routing, where every packet is sent and then relayed to everybody on the same frequency, is packet collisions.

Encryption

FreakWAN default mode of operation is as unencrypted everybody-can-talk open network. In this mode, messages can be spoofed, and different persons (nicks) or nodes IDs can be impersonated by other devices very easily.

For this reason it is also possible to use encryption with pre-shared symmetric keys. Because of the device limitations and the standard library provided by MicroPython, we had to use an in-house encryption mode based on AES and SHA256.

High level encryption scheme

Each device can store multiple symmetric keys, associated with a key name. Every time an encrypted message is received, all the keys are tested against the packet, and if a matching key is found (see later about the mechanism to validate the key) the message is correctly received, and displayed with the additional information of the key name, in order to make the user aware that this is an encrypted message that was decrypted with a specific key.

So, for example, if a key is shared between only two users, Alice and Bob, then Alice will store the xyz key with the name "BoB", ad Bob will store the same xyz key with the name "Alice". Every time Alice receives an encrypted message with such key, it will see:

#Bob bob> Hi Alice, how are you?

Where the first part is #<keyname>, and the rest of the message is the normal message, nick and text, or media type, as normally displayed.

Similarly if the same key is shared among a group of users, the effect will be to participate into a group chat.

Keys must be shared using a protected channel: either via messaging systems like Whatsapp or Signal, or face to face with the interested users. Optionally the system may decide to encrypt local keys using a passphrase, so that keys can't be extracted from the device when it is non operational.

Encrypted packets and algorithm

Only data messages are encrypted. ACKs, HELLO and other messages remain unencrypted.

The first four standard header fields of an encrypted packet are not encrypted at all: receivers, even without a key at hand, need to be able to check the message type and flags, the TTL (in case of relay), the message UID (to avoid reprocessing) and so forth. The only difference between the first 7 bytes (message type, flags, UID, TTL) of an ecrpyted and unencrypted message is that the Encr flag is set. Then, if such flag is set and the packet is thus encrypted, a 4 bytes initialization vector (IV) follows. This is the unencrypted part of the packet. The encrypted part is as the usual data as found in the DATA packet type: 6 bytes of sender and the data payload itself. However, at the end of the packet, there is an additional (not encrypted) 10 bytes of HMAC (truncated HMAC-256), used to check integrity, and even to see if a given key is actually decrypting a given packet correctly. The checksum computation is specified later.

This is the representation of the message described above:

+--------+---------+-------+-------+-------+-----------+--//--+------+
| type:8 | flags:8 | ID:32 | TTL:8 | IV:32 | sender:48 | data | HMAC |
+--------+---------+-------+-------+-------+-----------+--//--+------+
                                           |                  |
                                           `- Encrypted part -'

The 'IV' is the initialization vector for the CBC mode of AES, that is the algorithm used for encryption. However it is used together with all the unencrypted header part, from the type field, at byte 0, to the last byte of the IV. So the initialization vector used is a total 11 bytes, of which at least 64 bits of pseudorandom data.

The final 10 bytes HMAC is computed using HMAC-SHA256, truncated to 10 bytes, but the last 4 bits are set to the padding length of the message, so actually of the 80 bits HMAC, only the first 76 are used. The last byte least significant four bits tell us how many bytes to discard from the decrypted payload, because of AES padding.

The AES key and the HMAC key are different, but derived from the same master key. The derivation is performed as such: Given the unique key k, we derive the two keys with as:

aes-key = first 16 bytes of HMAC-SHA256(k,"AES14159265358979323846")
mac-key = HMAC-SHA256(k,"MAC26433832795028841971")

Encryption

To encrypt:

  1. Build the first four fields of the packet header as described above.
  2. Renerate a random 4 bytes IV and append it.
  3. Set TTL to 0 (but save the original value), and clear the Relayed flag bit, also saving the original flag value.
  4. Encrypt the payload with AES in CBC mode, using aes-key as key, and using as IV the first 11 bytes of the packet. Pad the packet with zeroes so that it is multiple of 16 bytes (AES block). Remember the padding used to reach the AES block size as "PADLEN". Append the encrypted payload to the packet.
  5. Perform HMAC-SHA256(mac-key,packet) of all the packet built so far.
  6. Append the first 10 bytes of the HMAC to the packet, but replace the last 4 bits with PADLEN as unsigned 4 bit integer.
  7. Restore TTL and Relayed flag.

Decrypting is very similar. When a packet arrives, we clear the TTL and Relayed bit (saving the original values). We also store the last 4 bits of the HMAC as PADLEN, and clear those bits. Then, for all the all the keys we have in memory, one after the other, we see if we can find one for which we can recompute the HMAC of the message (minus the last 10 bytes), trucate it to 10 bytes, clearing the last 4 bits, and check if it matches with the final 10 bytes of the packet.

If a match is found, we can decrpyt the message with AES in CBC mode, discard the final PADLEN zero bytes (checking they are actually zero as an additional check against implementation bugs), and process it.

Relaying of encrypted messages

The receiver of the packet has all the information required in order to relay the packet: we want the network to be collaborative even for messages that are not public. If the PleaseRelay flag is set, nodes should retransmit the message as usually, decrementing that TTL and setting the Relayed flag: the TTL and this flag are not part of the CHECKSUM computation, nor of the IV (they are set to 0), so can be modified by intermediate nodes without invalidating the packet. Similarly, the message UID is exposed in the unencrypted header, so nodes without a suitable key for the message can yet avoid re-processing the message just saving its UID in the messages cache.

Security considerations

Packets fragmentation (proposal not implemented)

LoRa packets are limited to 256 bytes. There is no way to go over such limitation (it is hardcoded in the hardware), and it would also be useless, since the long time on air means that, after a certain length, it is hard to take a good frequency reference: communication errors would be inevitable.

So the FreakWAN protocol supports fragmentation, for use cases when it is useful to transmit messages up to a few kilobytes at max. Fragmentation is only supported for DATA type message (the other message types don't need, to be larger than the LoRa packet size), and works both for clear text and encrypted messages.

In the sender size, when the total length of the packet would be more than MAXPACKET total bytes (that may be configured inside the app), the data payload is split in roughly equally sized packets. The last packet may have a byte more in case the data length is not multiple of the number of packets.

Important: MAXPACKET must be choosen so that it is always possible to transmit two bytes more than its value, since the data section will have two additional bytes used for the fragmentation metadata.

So, for instance, if the data section (nick + data o media) of a DATA packet is 1005 bytes, and MAXPACKET is set to 200 bytes, the number of total packets required is the integer result of the following division:

NUMPACKETS = (DATALEN+MAXPACKET-1) / MAXPACKET

That is:

NUMPACKETS = (1005+199)/200 = 6

Each of the packets will have the following size:

BASESIZE = 1005/6 = 167

However 167*6 = just 1002 bytes. So we also calculate a reminder size:

REM = DATALEN - BASESIZE*NUMPACKETS
REM = 1005 - (167*6) = 3

And we add a single additional byte of data to the first REM packets we generate during the fragmentation, so the length of the packets will be:

That is a total of 1005 bytes. The strategy used attempts at generating packets of roughly the same size, so that each packet has a better probability of being transmitted correctly. For our use case, to transmit the first N-1 full packets at the maximum length, and then a final small packet, would not be optimal.

Fragment header and data section

Each fragment will be sent as a DATA message is exactly like a normal DATA message, with the following differences:

  1. The Fragment flag is set in the header flags section.
  2. At the end of the fragment data, there are two 8 bit unsigned integers, appended to the data itself.

For instance, in the example above, the first packet would have 168 bytes of data terminated by two additional bytes:

//-----------------------+------------+-------------+
... 168 byts of data ... | frag-num:8 | tot-frags:8 |
/.-----------------------+------------+-------------+

Where frag-num is the number of the fragment, identifying its position among all the fragmnets, and tot-frags is the total number of fragments. The total length of the data is implicit, and is not direcly available.

The receiver will accumulate (for a given maximum time) message fragments having the same Message ID field value. Once all the fragments are received, the origianl message is generated from the fragments, where the Fragment flag is cleared, the data sections of all the fragments are glued together (but discarding the last two bytes of each packet), and finally the message is passed for processing to the normal FreakWAN path inside the FreakWAN stack. The software should make sure that fragments don't accumulate forever in case some fragment is missing: after a given time, if no full reassembly was possible, fargments should expire and the memory should be freed.