Proxy-Chosen Virtual Client Connection ID

[ehaydenr]

As described in #88, loop attacks are possible when clients pick the virtual connection ID. This change moves the responsibility of generating a Virtual Client Connection ID to the proxy and requires the proxy to generate unpredictable virtual connection IDs.

Unfortunately, this change complicates the capsule exchange. Specifically, the proxy cannot send forwarded mode packets in the Target->Client direction until it knows that the client is ready to receive them. Previously, when the client chose the vcid, we could require that the client not share the vcid unless it's ready to receive with it. Now that the proxy chooses the client vcid, we need the client to signal it's ready to receive forwarded mode packets. To accomplish this, ACK_CLIENT_VCID is introduced.

The ACK_CLIENT_VCID capsule solves the rule-readiness problem and maintains that the client can supply a Stateless Reset Token for resetting the client<->target tunnel.

[DavidSchinazi]

If the virtual client CID is chosen by the proxy, doesn't that mean that the client can no longer multiplex multiple connections on the same socket?

[ehaydenr]

If the virtual client CID is chosen by the proxy, doesn't that mean that the client can no longer multiplex multiple connections on the same socket?

I don't think so. The difference here is that the client has to create a demultiplexing rule with a value that the proxy chooses as opposed to one it chose itself. And the proxy can't send forwarded mode packets until the client signals that the rule is created.

Maybe I'm missing something, can you elaborate?

[DavidSchinazi]

In general the mindset of QUIC CIDs is that they're picked by the receiver. That allows among other things the receiver to pick their CID length (a length 1 might be sufficient for a client that's only multiplexing two connections, but a server multiplexing hundreds would want more bytes). If you let someone else pick the CIDs, you could end up with a conflict on the client if it's talking to two proxies and the first proxy picks CID=12345678 and the other picks CID=1234567890. One potential solution is to require a sufficient length that makes conflicts statistically unlikely, but the min length depends on how much multiplexing the client wants, so now maybe we need a way to encode that length?

[ehaydenr]

@DavidSchinazi what do you think about 98b876ef2ff61d02ec91722bd754c7e49ad696ad?

[DavidSchinazi]

I like that. We'll probably want to add some implementation considerations that

• tell proxies that they need to select these CIDs randomly (maybe even MUST?)
• tell clients that if they're multiplexing they need to pick a client connection ID length that makes conflicts unlikely if the proxy picks a random number for them

but in general this should work

[ehaydenr]

@DavidSchinazi, how does the language added in 4a2dc2164b69149ee54b2c76678d6791847907c0 sound?

[marten-seemann]

I have two concerns with this approach:

1. Unless I'm missing something, this design assumes that there's no CID-based routing happening inside the client's network. I don't feel comfortable hard-coding this assumption into the design here.
2. Similar to @DavidSchinazi, I'm concerned about the CID length here. My QUIC implementation assumes that all CIDs have the same (configurable) length. When a new packet arrives, I read n CID bytes, and find the respective QUIC connection by doing a map lookup. This doesn't work anymore if the proxy gives me a CID that has a different length. I'd either have to build some kind of prefix-trie, or do (up to 20-n) map lookups. Given that the CID lookup already shows up in CPU profiles during high-bandwidth transfers, both of these options seem suboptimal. Things get even more complicated if an implementation decides to encode the CID length into the CID itself.

[ehaydenr]

@marten-seemann, to address your second point, one option could be to change this from a SHOULD to a MUST

To reduce the likelihood of connection ID conflicts, the proxy SHOULD choose a Virtual Client Connection ID that is at least as long as the Client Connection

Your first point is well taken. For what it's worth, this requirement mirrors the requirement for the proxy - it must receive QUIC packets (with CIDs it did not choose) from the target and demultiplex them.

As mentioned at 119, I see three options for addressing the loop issue:

1. Prohibit VIP sharing
2. Eliminate target port sharing
3. Proxy-chosen Virtual Client CIDs (this PR)

While certainly not perfect, I think this option is the best of those three. Do you disagree? Do you see a another alternative?

[martinduke]

I have two concerns with this approach:

1. Unless I'm missing something, this design assumes that there's no CID-based routing happening inside the client's network. I don't feel comfortable hard-coding this assumption into the design here.

This is already a property of the design, before this PR, unless I'm missing something. Proxies are not able to choose the client CID that the target writes into packets, except for an ability to veto ones that collide. There is no ability to propose a routable connection ID for use on the target-facing interface.

[ehaydenr]

@martinduke, if I'm understanding @marten-seemann correctly, I think he's referring to the fact that, with this change, the client can no longer choose the virtual client connection IDs that it receives with in forwarding mode. I do agree with you that the proxy has never been able to influence the real client connection IDs beyond rejecting collisions - and this PR doesn't change that.

[martinduke]

@martinduke, if I'm understanding @marten-seemann correctly, I think he's referring to the fact that, with this change, the client can no longer choose the virtual client connection IDs that it receives with in forwarding mode. I do agree with you that the proxy has never been able to influence the real client connection IDs beyond rejecting collisions - and this PR doesn't change that.

I agree with your statement of the facts. But my point is that we've already negated the idea of connection-ID based load balancing on the client side. Since QUIC (modulo MPQUIC) doesn't allow changes to server address anyway, this is not much of a problem.

[ehaydenr]

@marten-seemann, thoughts?

[marten-seemann]

Sorry for the late response, I'll get back to this later today or tomorrow.

[marten-seemann]

2. Similar to @DavidSchinazi, I'm concerned about the CID length here. My QUIC implementation assumes that all CIDs have the same (configurable) length. When a new packet arrives, I read n CID bytes, and find the respective QUIC connection by doing a map lookup. This doesn't work anymore if the proxy gives me a CID that has a different length. I'd either have to build some kind of prefix-trie, or do (up to 20-n) map lookups. Given that the CID lookup already shows up in CPU profiles during high-bandwidth transfers, both of these options seem suboptimal. Things get even more complicated if an implementation decides to encode the CID length into the CID itself.

I spent some more time thinking about this point, and I'm wondering if I'm missing something here. Compared to a simple map lookup, looking up a variable-length CID is way slower. I implemented a prefix trie with (up to) 20 levels, and its performance is terrible. At 100k CIDs, a CID lookup takes up to 40x longer:

BenchmarkTrieLookup/4_byte_CIDs-16              20221346                64.73 ns/op
BenchmarkTrieLookup/8_byte_CIDs-16               5931970               207.9 ns/op
BenchmarkTrieLookup/15_byte_CIDs-16              2255654               523.1 ns/op
BenchmarkTrieLookup/20_byte_CIDs-16              1402430               805.1 ns/op
BenchmarkMapLookup/4_byte_CIDs-16               88730054                13.92 ns/op
BenchmarkMapLookup/8_byte_CIDs-16               91244919                13.14 ns/op
BenchmarkMapLookup/15_byte_CIDs-16              85435095                13.84 ns/op
BenchmarkMapLookup/20_byte_CIDs-16              41338683                29.12 ns/op

In these cases, it would be faster to do a map lookup for all 20 possible CID lengths, until a match is found.

Admittedly, this problem exists on the proxy side already, but now we're moving it to the client as well. There are probably faster algorithms than a prefix trie (prefix b-trees), but I'd be curious to learn what kind of data structure people are planning to use for this in practice.

[marten-seemann]

I'm not sure if this is a nice solution, but an alternative could be using a bloom filter. I described the idea in more detail in https://github.com/ietf-wg-masque/draft-ietf-masque-quic-proxy/issues/108.

[ehaydenr]

@marten-seemann, one way of avoiding variable length lookups is to use a minimum client cid size and reject conflicts. Let's say you want to use N-byte CIDs. You would send a REGISTER_CLIENT_CID capsule with your N-byte CID and receive a virtual client connection ID back that's N+M bytes long. For the purposes of connection lookup, just use the first N bytes. If the first N bytes conflicts with some other connection, don't proceed with forwarding mode or send another REGISTER_CLIENT_CID capsule to solicit a new virtual client connection ID.

[afressancourt]

Would it be possible to let the client choose between the method initially described in the draft and the mechanism proposed by @ehaydenr ? In the signaling, if the REGISTER_CLIEND_ID contains only a Client Connection ID, then we are letting the proxy propose the virtual Client CID, and if the REGISTER_CLIEND_ID contains both a Client Connection ID and a virtual Client Connection ID, the proxy tries to accommodate this request, and sends an error message if it is not possible to deal with a client-set virtual Client Connection ID ?

[martinduke]

Would it be possible to let the client choose between the method initially described in the draft and the mechanism proposed by @ehaydenr ? In the signaling, if the REGISTER_CLIEND_ID contains only a Client Connection ID, then we are letting the proxy propose the virtual Client CID, and if the REGISTER_CLIEND_ID contains both a Client Connection ID and a virtual Client Connection ID, the proxy tries to accommodate this request, and sends an error message if it is not possible to deal with a client-set virtual Client Connection ID ?

If I'm not mistaken, if the proxy allows the client to choose the connection ID, it is open to the attack that is the reason for this PR.

[afressancourt]

If I'm not mistaken, if the proxy allows the client to choose the connection ID, it is open to the attack that is the reason for this PR.

I understand the wish to mitigate loop attacks, but at the same time it complicates the connection setup with one additional message. Besides, we can argue that not answering the ACK_CLIENT_ID sent by the proxy can be used as an attack vector if clients coordinate to sent a high number of REGISTER_CLIENT_CID to the same proxy.

Letting the client choose whether he wants to let the proxy choose its virtual connection ID would allow to either ease the capsule exchange at connection setup, or fallback in a mode which protects the proxy against loop attacks.

[ehaydenr]

I understand the wish to mitigate loop attacks, but at the same time it complicates the connection setup with one additional message.

I would argue the additional message is cheap since it requires no additional round trip.

Besides, we can argue that not answering the ACK_CLIENT_ID sent by the proxy can be used as an attack vector if clients coordinate to sent a high number of REGISTER_CLIENT_CID to the same proxy.

The proxy can reject the CID registration with the corresponding CLOSE_CLIENT_CID/CLOSE_TARGET_CID capsule. For adding guidance here, I created #109

Letting the client choose whether he wants to let the proxy choose its virtual connection ID would allow to either ease the capsule exchange at connection setup, or fallback in a mode which protects the proxy against loop attacks.

Are you suggesting we keep the capsule exchange as-is and for deployments that wish to mitigate loop attacks, they simply don't implement forwarding mode?

[ehaydenr]

@marten-seemann , does my comment about skipping variable length lookups address your concern?

[marten-seemann]

@marten-seemann, one way of avoiding variable length lookups is to use a minimum client cid size and reject conflicts. Let's say you want to use N-byte CIDs. You would send a REGISTER_CLIENT_CID capsule with your N-byte CID and receive a virtual client connection ID back that's N+M bytes long. For the purposes of connection lookup, just use the first N bytes. If the first N bytes conflicts with some other connection, don't proceed with forwarding mode or send another REGISTER_CLIENT_CID capsule to solicit a new virtual client connection ID.

This makes sense to me. It assumes that there's not too much structure to CIDs issued (e.g. if the proxy's CID allocation strategy would be to use an N byte prefix, this method wouldn't work), but I think that's guaranteed by the fact that CIDs MUST be unlinkable to an on-path observer.

[afressancourt]

I understand the wish to mitigate loop attacks, but at the same time it complicates the connection setup with one additional message.

I would argue the additional message is cheap since it requires no additional round trip.

If you can send the ACK_CLIEN_VCID together with the first bytes of data, then, I agree, you "only" loose the overhead of the ACK message.

Letting the client choose whether he wants to let the proxy choose its virtual connection ID would allow to either ease the capsule exchange at connection setup, or fallback in a mode which protects the proxy against loop attacks.

Are you suggesting we keep the capsule exchange as-is and for deployments that wish to mitigate loop attacks, they simply don't implement forwarding mode?

Not exactly. I had in mind a mechanism in which the client starts with sending a REGISTER_CLIENT_ID with either a VCCID or not in it (or a bit stating the VCCID is proposed but can be overridden). If the proxy receives a REGISTER_CLIENT_ID with either no VCCID or an overriddable VCCID, then:

• It can propose a VCCID in the ACK_CLIENT_ID, which will be acked itself by the ACK_CLIENT_VCID.
• Otherwise, we are in a state in which the client does want to use its VCCID, so then either the proxy acks this sending a ACK_CLIENT_ID or it refuses the connection with the corresponding CLOSE_CLIENT_CID/CLOSE_TARGET_CID capsule.