bind error when server behind NAT

[ChristofKaufmann]

I encountered an error when trying to use owping, but the same holds for twping. The set up is illustrated here: The server reports the following error: "bind([PB]:PortT): Cannot assign requested address" (where I replaced the IP address and port to match the names in the figure).

I have some more information about the server state:

• It prints "Connection to ([IB]:PortC) from ([PA]:37618)" and similarly "ControlSession([IB]:PortC) accepted from userid(nil):([PA]:37618)" (where I replaced IP addresses and port again).
• The address IB is stored in cntrl->local_addr.
• The address PA is stored in cntrl->remote_addr.
• The address PB is stored in tsession->receiver (for the first session).
• The address IA is stored in tsession->sender (for the first session).

Now from my point of view it does not make any sense, that owampd tries to bind to PB instead to IB. tsession seems to hold the wrong addresses. This appears to be a bug.

I compiled it from current master (6140235f) on Ubuntu and Debian.

[vvidic]

I think this might be expected ad the protocol is not very NAT friendly and here we have not one but two NAT-s.

The client sends the both the sender and receiver address in the Request Session packet as it sees it - so receiver=PB and server tries to bind to that.

[ChristofKaufmann]

OK, but cntrl holds exactly the addresses that are required. Why does tsession hold different addresses?

[vvidic]

Control connection uses TCP so it passes through NAT, but each end sees the internal address for itself and public address for the other end:

• tsession is received from the client and the client observes IA:PB address pair
• ctrl is from the view of the server and the addresses are PA:IB.

UDP test stream is created using the client view tsession as sent in the Request Session control message. And even if IB was used on the server side the test would still probably fail because it would expect IA source address in the test packets and the NAT would rewrite that to PA. The solution would be to not include the addresses at all in the Request Session but I'm not sure that is possible at the moment.

[vvidic]

Some more info: https://tools.ietf.org/html/draft-muthu-ippm-twamp-nat-00

[vvidic]

TWAMP RFC mentions the option of zero address that might help in this case:

   The Sender Address and Receiver Address fields contain, respectively,
   the sender and receiver addresses of the endpoints of the Internet
   path over which a TWAMP-Test session is requested.  They MAY be set
   to 0, in which case the IP addresses used for the Control-Client to
   Server TWAMP-Control message exchange MUST be used in the test
   packets.

[ChristofKaufmann]

OK, thanks for the information. It makes sense to me now. How is that handled currently? Having a NAT is very common. Are there applications that can handle it using OWAMP? I mean, is the OWAMP API capable of handling it and is this therefore a problem of owping and/or owampd or does the API implementation require fixes?

Sending zero addresses sounds like a solution.

[vvidic]

NAT is currently unsupported and since OWAMP RFC does not mention zero addresses, the zero address fix might only be possible for TWAMP protocol.

[mfeit-internet2]

Now from my point of view it does not make any sense, that owampd tries to bind to PB instead to IB. tsession seems to hold the wrong addresses. This appears to be a bug.

It isn't a bug in the implementation, it's a behavior dictated by the protocol. As covered above, the description of the Request-Session message, RFC 4656 says:

The Sender Address and Receiver Address fields contain, respectively, the sender and receiver addresses of the end points of the Internet path over which an OWAMP test session is requested.

The receiver does not know that the address in the message is not one of its own until it tries to bind and fails. It also can't make assumptions about what interface might have been meant. The only way around it would be to give the server a translation table that says "disregard the protocol; when the receiver field says X, it should bind to Y instead."

You can get away with going through NAT on the sending side because the server implementation doesn't care that the test traffic is coming from somewhere else, so nothing breaks. (Now that I think about it, packets coming from addresses other than what was in the message should be rejected if the protocol is to be obeyed 100%. But there's also Postel's Law, which says you should be conservative in what you send and liberal in what you accept.)

[ChristofKaufmann]

Indeed no mention of zero addresses in OWAMP RFC. OWAMP was earlier proposed and also received less attention than TWAMP, it seems. Maybe it would also not make sense for OWAMP, since send session and receive session can be done sequentially and even separately. Even when a receive session follows immediately after a send session, UDP hole-punching might not persist the full receive session. So, maybe the RFC considered that already and this is the reason why a zero address is not mentioned there.

Regarding the last comment the OWAMP RFC, Section 6.2 says:

Unless otherwise configured, the default behavior of servers MUST be to decline requests where the Receiver Address field is not equal to the address that the control connection was initiated from or an address of the server (or an address of a host it controls).

This could be checked before binding, but I guess it is not important, when binding an endpoint to receive packets. The check should prevent DoS to third-parties, so it is more important when the server is sending.

To summarize the discussion up to here (please correct me if i'm wrong):

• For OWAMP there is no possibility to support servers behind NAT.
• For TWAMP it is possible and the protocol as defined in the RFC allows for NAT support.
• The implementation of twping and twampd do currently not support servers behind NAT.

[vvidic]

I would not say either of them support NAT because RFC does not mention NAT at all. Zero address might work with some types of NAT but is not a general solution for all NAT types. For NAT measurements a different protocol is likely needed.

[vvidic]

Seems owping even refuses to run tests when NAT is involved:

$ owping pstest2.geant.carnet.hr
owping: FILE=capi.c, LINE=796, Server denied test: pstest2.geant.CARNet.hr
owping: FILE=owping.c, LINE=228, Session Failed!

Server end reports:

Feb  6 11:17:34 pstest2 owampd[5062]: FILE=sapi.c, LINE=708, Test Denied: OpenMode recieve_addr(192.168.122.195) != control_client(161.53.12.222)

[vvidic]

TWAMP patch merged for 4.2.0 release.

[ThierryMajutec]

When trying to perform owping between two perfsonar docker instances, it works fine except when one instance sits behind a NAT router. I get the following error message: owping: FILE=capi.c, LINE=881, Server denied

Is using NAT still impossible ?

[ChristofKaufmann]

@ThierryMajutec I have used TWAMP behind NATs, but not with a recent version. OWAMP does not work with NATs. You have to use the -Z option for zero-address, have a look here: https://github.com/perfsonar/owamp/pull/50#issuecomment-460992637

[ThierryMajutec]

@ThierryMajutec I have used TWAMP behind NATs, but not with a recent version. OWAMP does not work with NATs. You have to use the -Z option for zero-address, have a look here: #50 (comment)

Thanks Christof. Unfortunately when using twping instead of owping, I get the following error twping: FILE=owping.c, LINE=1757, Unable to open control connection to...

[ChristofKaufmann]

As far as I know, the control connection should work (even without -Z), since the control connection is done via TCP and can handle NATs. That means the port forwarding is not set up correctly in the server-side NAT (or the server is not running). See the Figure in my original post at the top.