RFC: Uniform network stack integration

jia200x commented 4 years ago

Description

This PR summarizes the on-going efforts to provide a uniform experience when switching between different network stacks.

With this we should be able to:

Have the same support for network devices in all network stacks (if the network stack support that kind of devices). LWIP, OpenThread and GNRC support IEEE802.15.4 devices, but only GNRC support of all them.
Use the same API for sending/receiving data from the application in all network stacks.
Use the same API for configuring a network stack.

Architecture

+-------------------------------------------------------+
|                                                       |
|                       Upper layers                    |
|                                                       |
+-------------------------------------------------------+
            ^                                ^
            |                                |
           Sock                            Netif
            |                                |
            v                                v
+-------------------------------------------------------+
|                                                       |
|                       Network Stack                   |
|                                                       |
+-------------------------------------------------------+
            ^                                ^
            |                                |
    N.S South-bound API               N.S South-bound API
            |                                |
            v                                v
+----------------------+         +----------------------+
|                      |         |                      |
|     IEEE802.15.4     |         |       Ethernet       |
|     lower layer      |         |      lower layer     |
+----------------------+         +----------------------+

There are 3 key components that MUST be implemented by each network stack in order to have a uniform experience:

Sock

Depending on the supported layers (UDP, IPv6, IPv4), each network stack should implement sock_xxx members of the Sock API in order to be able to receive and send data from different layers (network, transport, etc). The Sock API should use the North-bound API of the network stack for that purpose.

Netif

The network stack should implement the Netif API in order to configure the network stack (e.g setting addresses, setting the interface up, set LoRaWAN keys, get TX power, etc)

South-bound API

Each network stack has a different South-bound API to access the link layers and network device drivers. Some network stacks require direct access to the network device (OpenWSN, OpenThread), while others require access to a Link Layer (GNRC, LWIP). Some network stacks have support for different link layers, while some others only for one (OpenWSN only supports IEEE802.15.4).

It's important to mention that South-Bound APIs are usually defined for a specific Link Layer technology or network device type. Some examples:

The current gnrc_netif_ieee802154 expects an IEEE802.154 SubMAC (#13376 ) to send/recv acknowledged data.
OpenWSN expects an IEEE802.15.4 Radio API

Current issues

If the netif identifier is independent of the network stack (https://github.com/RIOT-OS/RIOT/pull/12738), both the implementation of netif and sock API are straightforward because network stacks have defined APIs.

However, South Bounds API are experiencing 2 problems:

Some lower layers are not well divided:

E.g for IEEE802.15.4 radios , the netdev implementation mixes transceiver and link layer logic. This creates problems for stacks that require "radio operations" (prepare, transmit, read) instead of "link layer operations" (send). On the other hand, some stacks expect that the lower layer handle retransmissions, CSMA-CA and ACK features, but the behavior of the netdev API changes depending on the hardware accelerations present in the device.

Several tasks could be shared between South-bound API implementations

The reason why only a few set of radios are supported in LWIP and OpenThread is because each network stack has to initialize the radios and process their IRQ. Although a network stack is free to do so, these chores could be unified using e.g event_thread module or similar.

Roadmap

For each technology, define the boundaries and implement the missing pieces of each component of the lower network stack to make it compliant with South-Bound APIs.
- For example, the IEEE802.15.4 lower layers can be defined as a Radio Hardware Abstraction Layer, a SubMAC (#13376) and hopefully a full IEEE802.15.4 MAC layer.
Implement the South-Bound APIs with the components from 1.
Implement sock and netif in all network stacks

roberthartung commented 4 years ago

Thanks for the work. I hope to find some time looking at it and giving you feedback.

roberthartung commented 4 years ago

@jia200x We just had a longer discussion again about exchanging layers. Can we extend this issue with information about the contents of Network Stack in your architectural diagram? I know that it's a little out of scope, but I remember also talking to you about this. Do you know if there is another issue for that? I think it would be good to understand which layers are there and which APIs are used internally. We really are keen to use our own maclayers or netlayer.

Apart from that it would be helpful to clarify layer names.

EDIT:

From my understanding the Layers are well defined. Therefore you architectural picture should reflect these layers. IMO the network stack should include all layers. However in your picture there is at least the link layer not included, and the physical layer missing. I think it would be good to step back a bit and have a complete picture with all network stacks (including some internal details about the APIs), that features all layers.

roberthartung commented 4 years ago

WIP - Trying do define layers, and seeing what implementations are there.

+------------------------+
|      Application       |
+------------------------+
      ^            ^
      |            |
     Sock        Netif
      |            |
      v            v
+------------------------+
|                        | TCP
|        Transport       | UDP
|                        | 
+------------------------+

+------------------------+
|                        | IP
|         Network        | ICMP
|                        | 
+------------------------+

+------------------------+
|                        | 802.11 (WiFi)
|         Data Link      | 802.15.4 (IEEE 802.15.4)
|                        | 802.3 (Ethernet)
+------------------------+

+------------------------+
|                        | 1000BASE-T
|         Physical       | 802.11
|                        |  802.15.4
+------------------------+

jia200x commented 4 years ago

Hi @roberthartung

You are right with the layers approach. The thing is, I just tried to reflect how to integrate existing network stacks (OpenThread, LWIP, OpenWSN, GNRC) into RIOT. So, I just assumed that a "Network Stack" is a block with a southbound API that requires "some glue code" to interact with the OS and a northbound API to interact with the user.

However, we should indeed define entry points for each layer. A user should be able to access higher network stack layers (IP, UDP) as well as lower network stack layers (MAC, PHY).

jia200x commented 4 years ago

WIP - Trying do define layers, and seeing what implementations are there.

This is a good overview, thanks!

Some comments:

From the diagram, sock and netif would interface with several layers of the network stack. For instance, it's possible to access L3 with "sock_ip" or L4 with "sock_udp/sock_tcp". Same thing with netif (setting an IPv6 address is related to L3, and setting a PHY configuration is related to L1).
The interfaces between blocks are network stack specific (GNRC uses gnrc_netapi_t, OpenThread an internal API, etc). However, we can provide access to each layer that the network stack exposes and some others that are stack independent.

For instance, GNRC provides entry points to L2 (via gnrc_netif), L3 (gnrc_ipv6), L4 (gnrc_udp, gnrc_tcp) and higher layers (gcoap). We coud provide "full access" to each layer if we:

Implement the netif_t against gnrc_netif
Implement sock for higher network stack layers (gnrc_ipv6, gnrc_udp, etc).
Implement some entry points for lower network stack layers (MAC, PHY). We have traces of these (netdev), but we still don't have a way to access MAC layers (and it's still stack dependent)

jia200x commented 4 years ago

From my understanding the Layers are well defined. Therefore you architectural picture should reflect these layers. IMO the network stack should include all layers. However in your picture there is at least the link layer not included, and the physical layer missing. I think it would be good to step back a bit and have a complete picture with all network stacks (including some internal details about the APIs), that features all layers.

Sure, I will update the picture to be more specific

jia200x commented 4 years ago

Here's a follow up of the diagram to clarify what I meant (I'm not sure how to include netif_t there because it's not attached to a specific layer):

                       +------------------------+
                       |      Application       |
                       +------------------------+

                       +------------------------+
           sock_udp -> |                        | TCP
                       |        Transport       | UDP
                       |                        | 
                       +------------------------+

                       +------------------------+
            sock_ip -> |                        | IP
                       |         Network        | ICMP
                       |                        | 
                       +------------------------+

                       +------------------------+
       L2 interface -> |                        | 802.11 (WiFi)
                       |         Data Link      | 802.15.4 (IEEE 802.15.4)
                       |                        | 802.3 (Ethernet)
                       +------------------------+

                       +------------------------+
     PHY specific API-> |                        | 1000BASE-T
                       |         Physical       | 802.11
                       |                        | 802.15.4
                       +------------------------+

Some comments:

I think it might not make sense to provide PHY interfaces, because those layers are IMO way too specific to be abstracted. If someone requires to use a PHY layer (e.g to implement a custom MAC layer on top of IEEE802.15.4 compliant radios), then it should fallback to the PHY specific API. Note that most "raw link" applications (a.k.a sending data between radios) are L2.
It's important to mention that some layers have well known APIs (e.g 802.15.4 MAC and LoRaWAN use MLME/MCPS-SAP). However, we still need an interface on top of these APIs to send "technology independent L2 data". For instance, such a L2 interface could be used to send L2 data between Ethernet devices, 802.15.4, LoRaWAN, etc. Although they have different parameters, they have some stuff in common (addresses, etc). This is what we do with the txtsnd command, but it would be nice to have a {network stack, technology} independent mechanism

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

jia200x commented 3 years ago

side not to this: as discussed with @miri64 maybe it could be interesting to add some kind of L2 sockets (e.g sock_ieee802154, sock_lora, etc). Linux follows a similar approach for IEEE 802.15.4 devices. This would allow us to write stack independent code for LoRaWAN, CoAP over SLIP or any non-IP transport layer.

kaspar030 commented 2 years ago

The network stack should implement the Netif API in order to configure the network stack (e.g setting addresses, setting the interface up, set LoRaWAN keys, get TX power, etc)

Can we change that to use typed functions? That possibly just return "ENOTSUP"? I mean, netif_set_up(netif, ...) or netif_set_link(netif, NETIF_STATE_UP) vs netif_set_opt(...,..., NETIF_SET_UP, ...) or similar?

jia200x commented 2 years ago

Can we change that to use typed functions? That possibly just return "ENOTSUP"? I mean, netif_set_up(netif, ...) or netif_set_link(netif, NETIF_STATE_UP) vs netif_set_opt(...,..., NETIF_SET_UP, ...) or similar?

I think it's feasible and would probably force us to have a cleaner integration in the end. The only problem I see, is the fact we have too many divergent NETOPTs in the network interface that might end up in way too many functions.

On the other side, we could just define primitives for the network interface (up/down, MTU, l2 technology, etc) and start thinking of layer specific functions that receive a network interface.

E.g

/* To be implemented by every network stack */
netif_set_up(netif,...)

/* To be implemented by  all stacks that support 15.4 */
ieee802154_set_channel(netif, channel);
ieee802154_associate(netif,...);

/* To be implemented by all stacks that support LoRaWAN */
lorawan_set_nwkskey(netif,...);
lorawan_join_otaa(netif,...);

/* To be implemented by all stacks that support IPv6 */
ipv6_add_addr(netif, ...);

It might look tedious, but we have to do this anyway with a generic netif_set_op function. It also simplifies IMO the user documentation by orders of magnitude.

What do you think?

RIOT-OS / RIOT