search

openthread / ot-br-posix

OpenThread Border Router, a Thread border router for POSIX-based platforms.
https://openthread.io/
BSD 3-Clause "New" or "Revised" License
401 stars 226 forks source link

Increasing the IPv6 MTU #549

Closed INTENDRO closed 3 years ago

INTENDRO commented 4 years ago

I'm currently imlementing CoAPs nodes with OpenThread on Zephyr. The server certificate is larger than the 1280-byte MTU, gets fragmented and only the second and significantly smaller fragment is received by the nodes. I found this issue about messages larger than 1280. I setup some test application where I could test the MTU modifications using large UDP packets and succeeded to setup my nodes to send each other UDP messages with a maximum MTU of 2047 (as mentioned in this issue, this is limited by the 11bits for datagram length in the fragmentation header).

These are the OpenThread configurations that I changed in order to get this to run:

And this Zephyr-specific configuration (might not be of interest to you, but added for transparency):

Now I wanted to send these large UDP packets to the border router. First, I rebuilt the ot-rcp firmware for my nRF52840 dongle with the same configuration changes as for the nodes above. However, as it didn't work, I changed even more of then with the hope that one of them does the trick. These are the configuration changes for the ot-rcp firmware:

These configurations have also been used to recompile the border router firmware. Looking at the outputs from otbr-agent with journalctl, the border router receives UDP packets with a size of 1500 but cannot allocate a buffer large enough to save it:

Sep 01 17:41:51 otbr-agent[428]: [INFO]-MLE-----: Send Advertisement (ff02:0:0:0:0:0:0:1)
Sep 01 17:41:51 otbr-agent[428]: [INFO]-MAC-----: Sent IPv6 UDP msg, len:92, chksum:1400, to:0xffff, sec:no, prio:net
Sep 01 17:41:51 otbr-agent[428]: [INFO]-MAC-----:     src:[fe80:0:0:0:9c35:6e5c:9cf7:2f01]:19788
Sep 01 17:41:51 otbr-agent[428]: [INFO]-MAC-----:     dst:[ff02:0:0:0:0:0:0:1]:19788
Sep 01 17:41:53 otbr-agent[428]: [INFO]-MAC-----: Received IPv6 UDP msg, len:1548, chksum:138e, from:0xd800, sec:yes, prio:normal, rss:-59.625
Sep 01 17:41:53 otbr-agent[428]: [INFO]-MAC-----:     src:[****:****:****:****:9ae:40ff:e614:189f]:42771
Sep 01 17:41:53 otbr-agent[428]: [INFO]-MAC-----:     dst:[****:****:****:****:541f:5a8b:79d:84fd]:4444
Sep 01 17:41:53 otbr-agent[428]: [WARN]-PLAT----: processReceive: NoBufs
Sep 01 17:41:56 otbr-agent[428]: [INFO]-MLE-----: Send Advertisement (ff02:0:0:0:0:0:0:1)

I don't know how to go any further with this. Could someone please help me how to setup the borderrouter with a higher MTU.

Thanks a lot!

INTENDRO commented 4 years ago

After some trial and error I finally managed to get it to work. These are the necessary changes that I needed to apply in order for this to work as desired:

OpenThread nodes (Zephyr)

RCP connected to the border router (nRF52 Dongle)

Border router (Raspberry Pi 3B)

Increasing the MTU of the Network Interface This needs to be executed after every restart of the border router:

sudo ifconfig wpan0 mtu 2047 up

Used Versions

Commentary I think that setting the OPENTHREAD_CONFIG_IP6_MAX_DATAGRAM_LENGTH and OPENTHREAD_CONFIG_DEFAULT_SED_BUFFER_SIZE in order to increase the MTU makes sense. However, the VerifyOrExit command in ip6.cpp should take these config values into consideration as otherwise the packets just get dropped. The kMaxIp6Size constant was even harder to track down and two buffers (one in the function processReceive and one in processTransmit) get their size from this. I would argue that the size of these buffers should also get set by the configs and not be hardcoded.
Is it possible to set the interface MTU permanently instead of having to set it after every reboot using the ifconfig command mentioned above?

Irving-cl commented 3 years ago

Hi @INTENDRO, sorry for replying so late. Correct me if I'm wrong since I'm no expert for this part.

I think we can send a large message by using fragmentation. Current workaround seems to increase the max size of IP packets in order to send the entire UDP packet into one IP packet. Have you enabled the option OPENTHREAD_CONFIG_IP6_FRAGMENTATION_ENABLE in both of your builds as referred in PR#3948 ? Regarding VerifyOrExit in ip6.cpp:

// Do not forward reassembled IPv6 packets.
- VerifyOrExit(aMessage.GetLength() <= kMinimalMtu, error = OT_ERROR_DROP);
+ //VerifyOrExit(aMessage.GetLength() <= kMinimalMtu, error = OT_ERROR_DROP);

I think the purpose here is to avoid duplicates (mentioned in #4964). If we use fragmentation to handle a fragmented packet, the IPv6 fragments would be forwarded to the Host (ip6.cpp:911):

        case kProtoFragment:
            // Always forward IPv6 fragments to the Host.
            IgnoreError(ProcessReceiveCallback(aMessage, aMessageInfo, aNextHeader, aFromNcpHost, Message::kCopyToUse));

            SuccessOrExit(error = HandleFragment(aMessage, aNetif, aMessageInfo, aFromNcpHost));
            break;

And in current implementation, after the fragments are handled, ProcessReceiveCallback would be called again (HandleFragment -> HandleDatagram -> ProcessReceiveCallback). I think the intention is to reuse HandleDatagram to handle the reassembled message. And the reassembled message shouldn't be forwarded again. That's why VerifyOrExit(aMessage.GetLength() <= kMinimalMtu, error = OT_ERROR_DROP); is added.

Regarding the value of kMinimalMtu and kMaxIp6Size, I think 1280 for kMinimalMtu comes from RFC-2460:

  1. Packet Size Issues

    IPv6 requires that every link in the internet have an MTU of 1280 octets or greater.

I think it makes sense if a user wants to send a big IP packet without fragmentation and hence increase the max size of IP packets supported. So I agree to make kMinimalMtu and kMaxIp6Size calculated from a configurable variable. And for kMinimalMtu, we check that it's greater than 1280 by static assert.

So as a summary:

  1. I want to confirm that in your current workaround, is the big IP packet fragmented? If it's not, could you try to enable OPENTHREAD_CONFIG_IP6_FRAGMENTATION_ENABLE and see if it works. (I think we don't need to change other configuratations in this way.)
  2. I would go further to make all these Ip6 size related variables configurable so that it can support larger IP packet size with simple change in configurable.
INTENDRO commented 3 years ago

No problem. I was able to continue on my project as I was able to increase the MTU. So I wasn't completely stuck with this.

Just to give you some context why I wanted to increase the MTU instead of making use of the fragmentation: The person that gave me my current project told me to first get it to run with an increased MTU because they didn't get it to run with fragmentation. At this point it is important to mention, that the one that implemented the fragmentation in PR#3948 used to work at my current workplace.

We are working on a project using Zephyr RTOS and it seems that the fragmentation fails somewhere in Zephyr. If I flash two nodes with OpenThread and enable OPENTHREAD_CONFIG_IP6_FRAGMENTATION_ENABLE, I am able to send packets larger than 1280 bytes using the OpenThread CLI. However, my Zephyr application is not able to send packets that need to get fragmented. I don't want to bore you with Zephyr related problems but I just wanted to tell you why we are trying to increase the MTU instead of using fragmentation.

However, as the MTU is a configuration value, I would also expect to be able to increase it and not just having to use the fragmentation. Otherwise, the MTU could just be fixed to 1280 and then there wouldn't be a need for OPENTHREAD_CONFIG_IP6_MAX_DATAGRAM_LENGTH.

kMinimalMtu makes sense to me as the RFC-2460 clearly defines this value. Contrary to this I think that kMaxIp6Size should be set depending on the value in OPENTHREAD_CONFIG_IP6_MAX_DATAGRAM_LENGTH.

Irving-cl commented 3 years ago

@INTENDRO

Thanks for sharing the background. I think we can make the maximum Ip6 length configurable.

Besides, I'd like to know a little bit more about your test application. Is your test application something like a standalone udp server that listens on some address (maybe INADDR_ANY)? And the Ip6 packet would be forwarded to it through otbr-agent? Or it's integrated in otbr-agent?

In terms of supporting larger IP packet size, I think we can use another variable in VerifyOrExit(aMessage.GetLength() <= kMinimalMtu, error = OT_ERROR_DROP); and that variable is configurable. And we also make kMaxIp6Size configurable.

jwhui commented 3 years ago

Note that RFC 4944 specifies the IPv6 MTU as 1280 octets. Increasing kMinimalMtu can work if all devices within a given Thread network supports the larger kMinimalMtu value. However, if some devices do not support the larger MTU value, then your message may not be delivered.

Irving-cl commented 3 years ago

@jwhui

I think the purpose of this issue is to have some workaround to support a larger Ip6 packet, of course within a network where all devices support the larger size. To do this, currently it needs making small changes in the code instead of by simply providing a different configuration when building. Shall we provide a way to do this? I think this could exist as something 'non-official'. In real world we should always use the value 1280 while users can play with it by using values larger than that.

Regarding kMinimalMtu, I think this is an absolute constant (mentioned in RFC, not something configurable). I think the problem is that it shouldn't be used in some lines in current code if we allow configurable Max Ip6 size (for non-official use). Thoughts?

jwhui commented 3 years ago

I'm fine with parameterizing on OPENTHREAD_CONFIG_IP6_MAX_DATAGRAM_LENGTH. Just wanted to point out the danger of changing this value.

INTENDRO commented 3 years ago

@Irving-cl More info about my project:

Distributed nodes (built with Zephyr using OpenThread) send CoAP packets to a californium server. The border router receives the packets from the nodes and forwards them to a separate server (within our companies network) which hosts the californium server. Everything is encrypted using DTLS (therefore I should say that we use CoAPs instead of CoAP) and the certificates needed for this are pretty large. I mentioned that we currently cannot use the fragmentation because something does not work correctly within Zephyr. It's the DTLS handshake with the large certificates that gives us these problems.

As @jwhui said, we are aware of the fact that messages may not be delivered if some of the devices do not support the larger MTU. And I completely agree that making use of fragmentation is the way to go. The main goal of this project is to acquire more experience in the field of embedded security and I was told to go with the larger MTU as it was easier and faster to get working. Fragmentation will be looked at after the system is running in order to clean up the project and make it compatible with other devices that do not support the larger MTU.

bukepo commented 3 years ago

@INTENDRO did you make any progress of this issue?

jwhui commented 3 years ago

Closing stale issue.