'modern' SDPs from browsers too big for udp ng protocol

[olivermt]

Hi.

The standard MTU of 1500 is a bit small for modern SDPs with 3-4 streams each carrying 4-5 ICE candidates.

I know you can fragment datagrams across several frames with each having a MTU of 1500, but it is my understanding that there is no guarantee that we can control this across router configurations etc.

So my question is then, how feasible is it to get tcp as a transport for the ng protocol?

[olivermt]

We can of course just look for ice=force or similar flags in module and regexp-strip ice candidates before we ship, but in the (very rare) case that we want to only add ourself as a candidate and not replace all the others, then we have the same problem.

[rfuchs]

There are no plans for a TCP control protocol right now. Fragmentation is one of the core features of the IP protocol. If your network doesn't fragment packets properly, then you have more serious problems to worry about. Using TCP is not a solution for a broken network setup.

[olivermt]

Yes, I did more digging in the code of the opensips module itself and I agree. Something in the app stack is stopping sending of payloads above 1500MTU, and I think thats not how it should be.

Sorry for bothering you.

[etamme]

So I agree that if he is using RTPEngine's on a LAN, he ought to be able to properly control the setup to get packets to reassemble, or to turn some thing on like jumbo frames to prevent fragmentation in the first place, but in the case that RTPEngine is connected to off a LAN - this could be an issue.

[rfuchs]

If a high-MTU network is connected to a low-MTU network, then the intermediate router is responsible for fragmenting the packets. Alternatively the sending host can employ path MTU discovery. If either one of these doesn't work, then the network setup is broken.

[etamme]

Sure - and if its going across the public internet, there is likely not a way to fix it, which was my point. Im not disagreeing with you that IP fragmentation SHOULD work, I am just saying that practically speaking it is possible to run into issues quickly when sending large datagrams over the internet.

EDIT: And that WebRTC SDP's are almost certainly going to push packet sizes past to "standard" MTU's.

[rfuchs]

Routers on the public internet really shouldn't have these problems. And if you do run into fragmentation problems somewhere, then TCP will have the same problems. TCP can limit its segment size only after it has discovered the path MTU, and if PMTU discovery doesn't work, then TCP and UDP will both fail.

[etamme]

Yea - I am curious about rfuchs setup, and why he is getting issues b/c certainly in a controlled network, there should not be an issue.

[rfuchs]

Most fragmentation problems I've seen were either caused by cheap consumer-grade routers with broken IP stacks, or more recently by network drivers or hardware with broken fragmentation or checksum offloading. The latter can be disabled easily to fix the problem.

[etamme]

@olivermt I did a little test using netcat to send a 6000 byte UDP packet from my local machine to a digitalocean VM which was also listening (using netcat). I took a pcap of the transfer and saw that it recieved 3 1480 byte fragments which were automatically reassembled back to the 6k file on disk.

So ... yeah I think @rfuchs has a point that perhaps you have some issues with the network (ec2?) not doing proper frag/reassembly.

[olivermt]

I am getting about a 30-40% failure rate from AWS ireland datacentre to digital ocean amsterdam (very good peering) datacentre on anything bigger than the normal MTU size.

Do you have suggestions on what to do then? Because as it is, it is more or less unusable in production.

The rtpengine module for opensips (more or less verbatim ported from kamailio) uses sys/uio.h writev to send the iovector over the udp socket.

The OS on both ends is ubuntu 14.04.

I can almost certainly rule out a failure in the stack routers in the direction of opensips -> rtpengine, as our log ingestion system handles millions of messages every day where almost 40% of those are bigger than 1500 bytes (using syslog over UDP) living side by side in the same virtual private cloud on AWS.

I will do some netcatting to test further to see if that works from the same box to the same box. I'll set up a syntetic test of 1000 files of 3000 bytes and send them over the wire with netcat.

[rfuchs]

AWS is notorious for having problems with fragmentation.

[olivermt]

Really?

That is bad news.

I wonder why it works so well with our log ingestion system over UDP and this fails so much, we lose extremely few packets in that system.

Do you have any suggestions on what to do? Maybe refactor rtpengine module to try re-sending on failure? How do I prove that fragmentation reassembly is the perpetrator? I am not sure why I get the "message too big" error on some of them...

[rfuchs]

You can watch fragmentation in action by doing a packet capture (wireshark etc).

"Message too big" is a result of PMTU discovery, you should see that only once per destination and the message should go through at the next retransmission, which the rtpengine module (at least the Kamailio version) does do, even with subsecond intervals. No idea what the OpenSIPS module does in this regard.

A VPN might be a viable workaround for an unfixable broken IP stack. Other options are to try to play with offloading options, or simlpy to tell Amazon to fix their network.

I'm not opposed to TCP per se, but I don't see any reason to spend any time on it. If someone wants to contribute the necessary code, they're welcome to.

[olivermt]

I am guessing that the kamailio code does the retry, there seems to be nothing for that in opensips.

The loop that does the write is not listening for that return code, so it just bubbles upwards to generate an error is what I think is happening (I am a java developer, so C is not my strong side).

Anyhow I will see if I can modify the code to try again instantly by looking at what the kamailio module does.

It would be in rtpengines interest to be properly implemented on both those platforms I think (with websockets being a first class citizen in opensips 2.1, rtpengine is going to become the de-facto standard proxy engine for new setups using opensips I am pretty sure (not to mention that it seems far superior in general)).

I see you are cited as a contributor on both kamalio and the opensips module, so let me know if you dont mind me asking in this issue if I get stuck. I am already getting collab help from @etamme on #opensips @ freenode.

[rfuchs]

I've developed the Kamailio module, but I never had anything to do with the OpenSIPS module. This module was a port created by @lemenkov I believe.

[etamme]

fwiw, the relevant code in both projects rtpengine modules is nearly identical:

https://github.com/kamailio/kamailio/blob/master/modules/rtpengine/rtpengine.c#L1571

https://github.com/OpenSIPS/opensips/blob/master/modules/rtpengine/rtpengine.c#L1529

[olivermt]

I added the unhandled flag 'EMSGSIZE' to the while() statement and it seems to have fixed that it does not retry.

If it holds up under extended testing I will submit pull request to both projects.