Connection setup in quinn-proto

[jodiserla]

Hello, I was wondering how the connection setup works in quinn-proto. I am implementing Quic in a framework that has its own event loop and an udp_state which reads from, and writes to an udp socket. What I am trying to do is implement quinn-proto on top of that udp_state and triggering the read/write method when applicable. I have a bit hard time understanding how the connection setup works in quinn_proto. I start by calling the connection method listed in the endpoint, but its quite unclear to me what should happen afterwards - how can the other endpoint accept the connection? How do I know when both endpoints are connected? Are there any code examples, or docs I could check out to see how this should be done in quinn-proto?

[Ralith]

how can the other endpoint accept the connection?

Endpoint::handle will return a DatagramEvent::NewConnection when you pass it a datagram that successfully initiates a handshake.

How do I know when both endpoints are connected?

When Connection::poll returns Event::Connected.

Are there any code examples, or docs I could check out to see how this should be done in quinn-proto?

The API docs for quinn_proto::Connection will probably be of particular interest. In addition to the high level quinn crate itself, the test support code at https://github.com/quinn-rs/quinn/blob/main/quinn-proto/src/tests/util.rs might be a useful example.

[djc]

Note that since 0.9, the Quinn crate also has a runtime abstraction as well as an AsyncUdpSocket abstraction, so a viable alternative for your separate runtime/event loop use case might be to build on top of that instead of reinventing some of the wheels on top of quinn-proto.

[jodiserla]

I have been trying to follow examples from this https://github.com/quinn-rs/quinn/blob/main/quinn-proto/src/tests/util.rs here is my current attempt https://github.com/jodiserla/kompact/blob/1a002c42a64fd566662db39119a8f4ef2065125a/core/src/net/quic_endpoint.rs#L82 and here is how I call the methods to establish connection -I call the connect method in line 1484! https://github.com/jodiserla/kompact/blob/1a002c42a64fd566662db39119a8f4ef2065125a/core/src/net/network_thread.rs#L1476 My concerns include the amounts of calls I have to make to the read/write methods that both(shouldn't be both, but I am trying to figure out how everything works) call the drive method - which is similar to your examples drive method. Here is the output when I run the test case

Initiating Connection to ... 127.0.0.1:57878 push to outbound queue in read 127.0.0.1:57878 Nov 14 17:05:34.051 TRCE Writing quic from network thread 127.0.0.1:57876, addr: 127.0.0.1:57876, system: kompact-runtime-1, location: core/src/net/ network_thread.rs:487 THREAD2 SERVER READ QUIC n received 1200 OK in try read New connection Connection { handshake_cid: [121, 155, 96, 14, 9, 224, 57, 178] } push to outbound queue in read 127.0.0.1:57876 poll endpoint events EndpointEvent(NeedIdentifiers(Instant { t: 119627272806625 }, 4)) handle events ConnectionEvent(NewIdentifiers([IssuedCid { sequence: 1, id: [37, 96, 209, 34, 236, 250, 146, 213], reset_token: ResetToken([9, 114, 220, 199, 197, 245, 251, 239, 50, 246, 0, 86, 37, 3, 19, 2]) }, IssuedCid { sequence: 2, id: [82, 218, 173, 222, 144, 188, 212, 19], reset_token: ResetToken([53, 25, 95, 60, 82, 2, 228, 100, 131, 170, 24, 66, 25, 83, 10, 194]) }, IssuedCid { sequence: 3, id: [202, 53, 76, 214, 222, 190, 51, 239], reset_token: ResetToken([73, 25, 127, 152, 253, 121, 7, 167, 51, 31, 237, 69, 232, 69, 113, 186]) }, IssuedCid { sequence: 4, id: [45, 26, 93, 204, 185, 162, 229, 226], reset_token: ResetToken([194, 13, 125, 200, 27, 179, 112, 139, 119, 139, 223, 250, 213, 186, 14, 251]) }], Instant { t: 119627272806625 })) Nov 14 17:05:35.106 TRCE Reading quic from network thread 127.0.0.1:57878, addr: 127.0.0.1:57878, system: kompact-runtime-2, location: core/src/net/ network_thread.rs:472 push to outbound queue in read 127.0.0.1:57876 timeout Connection { handshake_cid: [121, 155, 96, 14, 9, 224, 57, 178] } n received 1200 OK in try read Redirect to existing connection ConnectionHandle(0) Nov 14 17:05:36.112 TRCE Writing quic from network thread 127.0.0.1:57878, addr: 127.0.0.1:57878, system: kompact-runtime-2, location: core/src/net/ network_thread.rs:487 timeout Connection { handshake_cid: [65, 200, 219, 167, 12, 18, 140, 32] } Nov 14 17:05:36.123 TRCE Reading quic from network thread 127.0.0.1:57876, addr: 127.0.0.1:57876, system: kompact-runtime-1, location: core/src/net/ network_thread.rs:472 push to outbound queue in read 127.0.0.1:57878 poll endpoint events EndpointEvent(ResetToken(127.0.0.1:57878, ResetToken([146, 206, 160, 190, 254, 86, 50, 137, 222, 196, 68, 254, 76, 26, 78, 188]))) poll endpoint events EndpointEvent(NeedIdentifiers(Instant { t: 119628326543500 }, 4)) handle events ConnectionEvent(NewIdentifiers([IssuedCid { sequence: 1, id: [185, 85, 251, 138, 98, 189, 111, 228], reset_token: ResetToken([11, 102, 162, 59, 119, 51, 113, 81, 150, 142, 35, 225, 43, 161, 242, 73]) }, IssuedCid { sequence: 2, id: [225, 117, 88, 15, 21, 123, 18, 50], reset_token: ResetToken([205, 223, 78, 205, 135, 248, 177, 173, 102, 56, 123, 169, 107, 135, 189, 162]) }, IssuedCid { sequence: 3, id: [4, 167, 246, 247, 144, 49, 73, 101], reset_token: ResetToken([86, 49, 102, 27, 246, 0, 89, 176, 31, 227, 204, 39, 54, 70, 250, 88]) }, IssuedCid { sequence: 4, id: [172, 23, 99, 175, 55, 206, 20, 220], reset_token: ResetToken([215, 96, 159, 231, 163, 143, 237, 165, 69, 151, 147, 24, 2, 114, 85, 166]) }], Instant { t: 119628326543500 })) Nov 14 17:05:37.124 TRCE Writing quic from network thread 127.0.0.1:57876, addr: 127.0.0.1:57876, system: kompact-runtime-1, location: core/src/net/ network_thread.rs:487 n received 1200 OK in try read Redirect to existing connection ConnectionHandle(0) push to outbound queue in read 127.0.0.1:57876 push to outbound queue in read 127.0.0.1:57876 timeout Connection { handshake_cid: [121, 155, 96, 14, 9, 224, 57, 178] } Nov 14 17:05:38.129 TRCE Reading quic from network thread 127.0.0.1:57878, addr: 127.0.0.1:57878, system: kompact-runtime-2, location: core/src/net/ network_thread.rs:472 test net::network_thread::tests::establish_connection_quic

This is for a master thesis where I am investigating low latency transport protocols in actor systems(the one being used is called kompact and has been developed by distributed computing group at KTH), my supervisor didn't want me to use tokio and therefore we chose quinn-proto. I will discuss with him further about quinns runtime abstraction and see if that might be a better idea since i have been struggling understanding what is going on in quinn-protos methods. In the meantime I would appreciate any feedback on this!

On Sun, 13 Nov 2022 at 21:04, Dirkjan Ochtman @.***> wrote:

Note that since 0.9, the Quinn crate also has a runtime abstraction as well as an AsyncUdpSocket abstraction, so a viable alternative for your separate runtime/event loop use case might be to build on top of that instead of reinventing some of the wheels on top of quinn-proto.

— Reply to this email directly, view it on GitHub https://github.com/quinn-rs/quinn/issues/1441#issuecomment-1312809660, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFY7T3RFVVOCPXJJAW74SVTWIFCVRANCNFSM6AAAAAAR7E6XGA . You are receiving this because you authored the thread.Message ID: @.***>

[jodiserla]

I feel like the problem is partly from the NeedIdentifiers and NewIdentifiers, but I am not 100% sure..

On Mon, 14 Nov 2022 at 17:13, Jódís Erla @.***> wrote:

I have been trying to follow examples from this https://github.com/quinn-rs/quinn/blob/main/quinn-proto/src/tests/util.rs here is my current attempt https://github.com/jodiserla/kompact/blob/1a002c42a64fd566662db39119a8f4ef2065125a/core/src/net/quic_endpoint.rs#L82 and here is how I call the methods to establish connection -I call the connect method in line 1484! https://github.com/jodiserla/kompact/blob/1a002c42a64fd566662db39119a8f4ef2065125a/core/src/net/network_thread.rs#L1476 My concerns include the amounts of calls I have to make to the read/write methods that both(shouldn't be both, but I am trying to figure out how everything works) call the drive method - which is similar to your examples drive method. Here is the output when I run the test case

Initiating Connection to ... 127.0.0.1:57878 push to outbound queue in read 127.0.0.1:57878 Nov 14 17:05:34.051 TRCE Writing quic from network thread 127.0.0.1:57876, addr: 127.0.0.1:57876, system: kompact-runtime-1, location: core/src/net/ network_thread.rs:487 THREAD2 SERVER READ QUIC n received 1200 OK in try read New connection Connection { handshake_cid: [121, 155, 96, 14, 9, 224, 57, 178] } push to outbound queue in read 127.0.0.1:57876 poll endpoint events EndpointEvent(NeedIdentifiers(Instant { t: 119627272806625 }, 4)) handle events ConnectionEvent(NewIdentifiers([IssuedCid { sequence: 1, id: [37, 96, 209, 34, 236, 250, 146, 213], reset_token: ResetToken([9, 114, 220, 199, 197, 245, 251, 239, 50, 246, 0, 86, 37, 3, 19, 2]) }, IssuedCid { sequence: 2, id: [82, 218, 173, 222, 144, 188, 212, 19], reset_token: ResetToken([53, 25, 95, 60, 82, 2, 228, 100, 131, 170, 24, 66, 25, 83, 10, 194]) }, IssuedCid { sequence: 3, id: [202, 53, 76, 214, 222, 190, 51, 239], reset_token: ResetToken([73, 25, 127, 152, 253, 121, 7, 167, 51, 31, 237, 69, 232, 69, 113, 186]) }, IssuedCid { sequence: 4, id: [45, 26, 93, 204, 185, 162, 229, 226], reset_token: ResetToken([194, 13, 125, 200, 27, 179, 112, 139, 119, 139, 223, 250, 213, 186, 14, 251]) }], Instant { t: 119627272806625 })) Nov 14 17:05:35.106 TRCE Reading quic from network thread 127.0.0.1:57878, addr: 127.0.0.1:57878, system: kompact-runtime-2, location: core/src/net/ network_thread.rs:472 push to outbound queue in read 127.0.0.1:57876 timeout Connection { handshake_cid: [121, 155, 96, 14, 9, 224, 57, 178] } n received 1200 OK in try read Redirect to existing connection ConnectionHandle(0) Nov 14 17:05:36.112 TRCE Writing quic from network thread 127.0.0.1:57878, addr: 127.0.0.1:57878, system: kompact-runtime-2, location: core/src/net/ network_thread.rs:487 timeout Connection { handshake_cid: [65, 200, 219, 167, 12, 18, 140, 32] } Nov 14 17:05:36.123 TRCE Reading quic from network thread 127.0.0.1:57876, addr: 127.0.0.1:57876, system: kompact-runtime-1, location: core/src/net/ network_thread.rs:472 push to outbound queue in read 127.0.0.1:57878 poll endpoint events EndpointEvent(ResetToken(127.0.0.1:57878, ResetToken([146, 206, 160, 190, 254, 86, 50, 137, 222, 196, 68, 254, 76, 26, 78, 188]))) poll endpoint events EndpointEvent(NeedIdentifiers(Instant { t: 119628326543500 }, 4)) handle events ConnectionEvent(NewIdentifiers([IssuedCid { sequence: 1, id: [185, 85, 251, 138, 98, 189, 111, 228], reset_token: ResetToken([11, 102, 162, 59, 119, 51, 113, 81, 150, 142, 35, 225, 43, 161, 242, 73]) }, IssuedCid { sequence: 2, id: [225, 117, 88, 15, 21, 123, 18, 50], reset_token: ResetToken([205, 223, 78, 205, 135, 248, 177, 173, 102, 56, 123, 169, 107, 135, 189, 162]) }, IssuedCid { sequence: 3, id: [4, 167, 246, 247, 144, 49, 73, 101], reset_token: ResetToken([86, 49, 102, 27, 246, 0, 89, 176, 31, 227, 204, 39, 54, 70, 250, 88]) }, IssuedCid { sequence: 4, id: [172, 23, 99, 175, 55, 206, 20, 220], reset_token: ResetToken([215, 96, 159, 231, 163, 143, 237, 165, 69, 151, 147, 24, 2, 114, 85, 166]) }], Instant { t: 119628326543500 })) Nov 14 17:05:37.124 TRCE Writing quic from network thread 127.0.0.1:57876, addr: 127.0.0.1:57876, system: kompact-runtime-1, location: core/src/net/ network_thread.rs:487 n received 1200 OK in try read Redirect to existing connection ConnectionHandle(0) push to outbound queue in read 127.0.0.1:57876 push to outbound queue in read 127.0.0.1:57876 timeout Connection { handshake_cid: [121, 155, 96, 14, 9, 224, 57, 178] } Nov 14 17:05:38.129 TRCE Reading quic from network thread 127.0.0.1:57878, addr: 127.0.0.1:57878, system: kompact-runtime-2, location: core/src/net/ network_thread.rs:472 test net::network_thread::tests::establish_connection_quic

This is for a master thesis where I am investigating low latency transport protocols in actor systems(the one being used is called kompact and has been developed by distributed computing group at KTH), my supervisor didn't want me to use tokio and therefore we chose quinn-proto. I will discuss with him further about quinns runtime abstraction and see if that might be a better idea since i have been struggling understanding what is going on in quinn-protos methods. In the meantime I would appreciate any feedback on this!

On Sun, 13 Nov 2022 at 21:04, Dirkjan Ochtman @.***> wrote:

Note that since 0.9, the Quinn crate also has a runtime abstraction as well as an AsyncUdpSocket abstraction, so a viable alternative for your separate runtime/event loop use case might be to build on top of that instead of reinventing some of the wheels on top of quinn-proto.

— Reply to this email directly, view it on GitHub https://github.com/quinn-rs/quinn/issues/1441#issuecomment-1312809660, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFY7T3RFVVOCPXJJAW74SVTWIFCVRANCNFSM6AAAAAAR7E6XGA . You are receiving this because you authored the thread.Message ID: @.***>

[Ralith]

You need to keep driving the connection until there's no more work to do. Often, more work is generated while driving the connection, e.g. due to endpoint events or timeouts, so multiple passes may be required.