gateway: The maximum frame size is calculated too aggressively

[shitz]

When calculating the maximum frame size the Gateway currently uses the path MTU given by the Daemon and subtracts the lengths of the common, address, path, and overlay headers. However, this relies on the path MTU returned by the Daemon to be very accurate. The Daemon itself bases the path MTU on information in the beacons which can at most be taken as a hint. A single misconfiguration can lead to the metric being off.

In the medium- to long-term the Gateway should do path MTU discovery as part of the monitoring of a path. In the short-term we should fix the maximum frame size to a conservative value, e.g., 1200 bytes.

[kormat]

SCIOND itself bases the path MTU on information in the beacons which can at most be taken as a hint

My view is that either we enforce this MTU to be correct, or we drop it. I don't see the point in having one but not be able to rely on it.

[jiceatscion]

@shitz Do you think it is the job of the Gateway to do MTU discovery, as opposed to the job of the Daemon? After all, the API seems to be that the Daemon provides paths with MTU. So, if that MTU needs to be more accurate, should that not be addressed in the Daemon?

Continuing along those lines, if the Daemon gets its info from the beacon, and if the Daemon needs to provide a better MTU; shouldn't the MTU provided by the beacon be more accurate? In fact, isn't beaconing the most economical opportunity to do MTU discovery?

If the answer to these questions is "NO", then it is pointless to include an MTU in beacons, and it is pointless to include an MTU in the segments that the daemon returns. We can just throw it all away and replace it with whatever guess the Beacons contain, directly at the point of use; for the same result; pending the implementation of some proper MTU discovery service.

WDYT?

[shitz]

I agree with the sentiment of @kormat here, i.e., either the MTU value in the beacon should be reliable or then it shouldn't be there at all. However, even if they are reliable there is a challenge for the MTU in the source and destination ASes. There is only a single MTU value that can be included in the beacon for intra-AS MTU, however, that might not be uniform throughout an AS. For example, consider endhosts in a big ISPs network. There will be various kinds of middleboxes or encapsulations and not all of them might be the same for every part of the internal network. For intermediate ASes this could also be a problem, however, there the network between the SCION routers should be much more uniform.

To solve that you you rely simply on the path MTU discovery mechanisms of the transport protocol and if your application uses a transport protocol that doesn't have that, then it must do its own path MTU discovery (that would be the case for the SIG).

After 5 more years of practical experience I have to say that using the MTU values contained in the beacons have worked well for us, but keep in mind that is mostly SIG-usage and those SIGs are usually placed at the border of a network without interfering middleboxes. It seems that the current SCION operator community sets the MTU values in their beacons reliably.

So while we didn't adopt the conservative value as proposed in the issue, we also didn't implement a path MTU discovery in the SIG. The issue can be closed for all I care.

[jiceatscion]

Trying to make progress on this, with the IETF drafts in mind: we can't be quite as flippant there as we are here.

So, correct me if I'm wrong:

• The issue of variable MTU is mostly a gateway problem: I take it that the gateway might change the path taken by a regular UDP connection , thereby invalidating the sender's assumptions.
• Besides gateway'ed traffic, the MTU of a connection should not have to change since it is associated with a path; unless the MTU of some link of the path changes during the life-time of the connection.

For the first case we could say: "tough luck; the gateway responds with an ICMP message, or fragments datagrams." For the second case, could we decide on something? For example:

• The MTU of a segment is the MIN of the MTUs of all the links in both directions.
• The MIN is calculated as the beacon propagates.
• The MTU of a segment cannot be changed. If one of the links reduces its MTU, a router will gets an ICMP error and will react by sending an SCMP "packet too big" error... until the segment expires.
• If an application gets such an error it can fall back to reducing its datagram size to 1200.

Thoughts?

[jiceatscion]

After taking a closer look at the mechanics... we have to specify something in the ietf drafts. Indeed the accuracy of paths MTUs is at the mercy of a miss-configuration but is otherwise predictible, if un-optimal:

There are, I think, only two issues:

• Misconfiguration
• Low global intra-AS MTU unnecessarily impacting some hops between two distinct BRs of an AS

These are correctible:

• First, without changing the protocol, it would be possible for a CS to reduce the ingress_mtu of a hop during beaconing, so that the hop's MTU is less-than-or-equal-to the MTU of the path between the two BRs involved. Once that feature is globally available (ok, so, in a way that's a change of protocol but it is bw compatible), a client that builds a path no-longer needs to constrain the path's MTU by the MTUs of every AS along the path; it can basically ignore them all (except the first and last). This would allow a large MTU between specific BRs to be taken advantage of, even if the lowest MTU of the AS is small.
• Second, given that, it becomes advantageous to discover the MTU between BRs; that is up to the CS implementation; the protocol is unaffected.
• Finally, we would still depend on configuration to inform the AS' global MTU (unless some costly global probing is used), but an overly conservative setting would be less damageable.

In the drafts I have to be honest with the current state of affairs. So I propose to mention that this is a possibility but is not the current practice. See: https://github.com/scionassociation/scion-dp_I-D/issues/39#issuecomment-2331114545 and https://github.com/scionassociation/scion-cp_I-D/issues/57#issuecomment-2358045291