PVA Server's use of random 'sendTransport' UDP port makes it impractical for gateway

[kasemir]

One key use case for a CA or PVA gateway is enabling read access to the control system through firewalls. Inside the control system network, servers and clients use broadcasts to find each other. From the outside, only very few UDP and TCP ports are open, limiting access to just a gateway.

When using the P4P gateway, which is based on the pvAccessCPP server, such a setup is nearly impossible because the gateway's port usage can't be controlled.

When starting a pvAccessCPP server with 'EPICS_PVAS_SERVER_PORT=5077', the server will listen to clients on UDP 5077 and then also serve data via TCP 5077, so you can open UDP and TCP 5077 on the firewall. But whenever the server receives a client's search request on UDP 5077, the server sends the "I have that channel, talk to me on TCP 5077" message from an arbitrary UDP port, not UDP 5077. With only UDP and TCP 5077 allowed through the firewall, the client never sees the reply. On option to make this work is configuring the firewall to allow all outgoing UDP traffic from the server, but that's counter the purpose of a firewall.

The reason for this pvAccessCPP server behavior is in the setup of its sendTransport. With a very simple patch, the server would use the configured UDP port for the sendTransport, so search requests received by the configured port result in a reply from that same port, firewall configuration is easy and based on preliminary tests it all "works":

diff --git a/src/remote/blockingUDPTransport.cpp b/src/remote/blockingUDPTransport.cpp
index 16b377a..d25b161 100644
--- a/src/remote/blockingUDPTransport.cpp
+++ b/src/remote/blockingUDPTransport.cpp
@@ -594,7 +594,7 @@ void initializeUDPTransports(bool serverFlag,
     osiSockAddr anyAddress;
     memset(&anyAddress, 0, sizeof(anyAddress));
     anyAddress.ia.sin_family = AF_INET;
-    anyAddress.ia.sin_port = htons(0);
+    anyAddress.ia.sin_port = htons(listenPort);
     anyAddress.ia.sin_addr.s_addr = htonl(INADDR_ANY);

     sendTransport = connector.connect(responseHandler, anyAddress, protoVer);

There may, however, be dragons. Issue #128 mentions problems with sharing UDP ports. Michael cautioned negative side effects related to the CMD_ORIGIN_TAG handling, resulting in message loops.

Who else is using the PVA gateway? Is your firewall wide open to allow for the random sendTransport port? Can you test with this patch?

[mdavidsaver]

@bhill-slac What is the situation at SLAC wrt. firewall configuration?

[bhill-slac]

SLAC doesn't allow any CA or PVA thru the firewall. We have specific host machines that allow ssh connections from outside the firewall and a VPN that can allow access to SLAC's network. From there you can ssh to hosts inside the control system networks and all CA or PVA must originate from within the control system networks. Our gateways are configured to allow read and some write access between control system subnets.

[kasemir]

So between the PVA gateway and clients, do you basically have all UDP ports open since the PVA gateway will reply to name searches via unknown ports?

[bhill-slac]

Yes, we keep most if not all ports open between control subnets.

On 3/30/2020 5:37 AM, Kay Kasemir wrote:

So between the PVA gateway and clients, do you basically have all UDP ports open since the PVA gateway will reply to name searches via unknown ports?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/epics-base/pvAccessCPP/issues/159#issuecomment-605973611, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABPDOJVVPRNYHB4TGWFBWFTRKCHCLANCNFSM4LOTJIUA.

[anjohnson]

A major usage of CA Gateways at APS is to provide read-only access between otherwise firewalled subnets, which I thought was similar to the SNS setup. However we run our gateways on VMs that have network interfaces on both sides of the firewall, so search requests to the gateway don't have to pass through the firewall at all, but it sends out its own search requests through the other interface. @kasemir Is your gateway configuration different? It sounds like you have a firewall hole that lets searches through but only when they are directed at the specific gateway machine and port. I think our setup would work fine with the existing PVA gateway behavior.

[kasemir]

At the SNS, control system subnets (VLANs) don't need/use a gateway. All subnets are simply listed in the ..ADDR_LIST. Gateways are used to get read-only access from the office network to the controls network, through a firewall/dmz/.. So there's a very small number of firewall openings (UDP search port, TCP data port). The firewall allows searches to reach the gateway's UDP search port, and the gateway can then reply from that port back to the client. The client can then read data via the gateway's TCP data port.

With the current PVA gateway (actual C++ PVA server code), the problem is that yes, the client can reach the gateway's UDP search port, but the PVA server then replies via a random UDP port, and that's blocked unless we opened all outgoing UDP traffic from the PVA server, which defeats the purpose of a firewall.

[ralphlange]

So I guess we need search on TCP, right? Just like for Channel Access.

[kasemir]

Sure, going the larger step of also supporting search via TCP is a separate option with some other benefits.

Still, I'd consider the current C++ PVA server implementation as simply broken. Why would a search reply received by EPICS_PVAS_SERVER_PORT get sent from a random port? Both a stateful firewall and most humans would expect the reply to X -> Y come as X <- Y, not X <- Z.

[mdavidsaver]

First, I have to plug PVXS where the server always replies to UDP searches through the receiving port. (I can't claim to have foreseen this effect, but I've never liked how pvAccessCPP handled UDP search)

Second, pvAccessCPP has (sort of) search by TCP. It's handled differently than with CA, and doesn't work completely (monitors don't reconnect). With PVXS, I'm tempted to jettison the existing approach and go with something much closer to CA.

Third, and there is also the possibility of multicast search.

[ralphlange]

Replying from the same port sounds reasonable, and would even be backward compatible as a protocol change, correct? I can't see how a client would be relying on the answer coming from a random port. Could be the same port out of chance now.

[mdavidsaver]

Why would a search reply received by EPICS_PVAS_SERVER_PORT get sent from a random port?

I completely agree. I don't know why this particular design was chosen for pvAccessCPP. It may just be (another) question of interface. The code processing search requests ServerSearchHandler::handleResponse() has access to the receiving socket (the transport argument). However, the code which sends the replyServerChannelFindRequesterImpl::channelFindResult() does not, and so instead uses a different socket.