search

quinn-rs / quinn

Async-friendly QUIC implementation in Rust
Apache License 2.0
3.76k stars 380 forks source link

Problem with establishing connection #1694

Closed serzhiio closed 11 months ago

serzhiio commented 11 months ago

I am experiencing some problems after i moved my project's quic-server to cloud server from dev machine. Client is able to establish connection but it dropped after several seconds. Here is client log:

[2023-10-21T13:41:01.786Z INFO  thread::eventloop::iouring::uni::quic::quinn::instance] Preparing connection to server! Address: 51.195.*.*:4474, domain: `51.195.*.*`.
[2023-10-21T13:41:01.786Z TRACE quinn_proto::endpoint] initial_dcid=ccfb96b79b0f7281c0e584ec8cfd6bc6ae967c38
[2023-10-21T13:41:01.788Z TRACE quinn_proto::connection] wrote 267 Initial CRYPTO bytes
[2023-10-21T13:41:01.788Z INFO  thread::eventloop::iouring::uni::quic::quinn::instance] ClientConnection(t:0|51.195.*.*:4474|51.195.*.*|) is established! Handle: ConnectionHandle(0)
[2023-10-21T13:41:01.788Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=0
[2023-10-21T13:41:01.788Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:01.788Z TRACE quinn_proto::connection::packet_builder] PADDING * 874
[2023-10-21T13:41:01.788Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection] got Initial packet (116 bytes) from 51.195.*.*:4474 using id 01544ef244f629e8
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection] recv; space=Initial
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection] retrying with CID 0bcb47e199f7c281
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection] discarding Initial keys
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=1
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:01.926Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection] PTO fired in_flight=1200 count=0 space=Initial
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=2
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=3
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:02.926Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] got Initial packet (1200 bytes) from 51.195.*.*:4474 using id 01544ef244f629e8
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] recv; space=Initial pn=1
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] switching remote CID to 2c99c6ca4bea5319
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] frame; ty=CRYPTO
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] consumed 90 CRYPTO bytes
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] Handshake keys ready
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=4
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] ACK ArrayRangeSet([1..2]), Delay = 0us
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection::packet_builder] PADDING * 1071
[2023-10-21T13:41:03.064Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection] PTO fired in_flight=4800 count=1 space=Initial
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=5
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=6
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:04.925Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection] PTO fired in_flight=7200 count=2 space=Initial
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=7
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=8
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-21T13:41:08.921Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:14.925Z TRACE quinn_proto::connection] timeout timer=Idle
[2023-10-21T13:41:14.925Z TRACE quinn_proto::connection] connection closed
[2023-10-21T13:41:14.925Z WARN  thread::eventloop::iouring::uni::quic::quinn::instance] Client evt: Connection lost: timed out!
[2023-10-21T13:41:14.926Z INFO  thread::eventloop::iouring::uni::quic::quinn::instance] Preparing connection to server! Address: 51.195.*.*:4474, domain: `51.195.*.*`.
[2023-10-21T13:41:14.926Z TRACE quinn_proto::endpoint] initial_dcid=34ec7acd417ad40fcc06cd05be0f2ef0ace2c987
[2023-10-21T13:41:14.926Z TRACE quinn_proto::connection] wrote 267 Initial CRYPTO bytes
[2023-10-21T13:41:14.926Z WARN  thread::eventloop::iouring::uni::quic::quinn::instance] Drop QConn:0 | local_uni_chs: 0, local_bidi: 0, remote_uni: 0, remote_bidi: 0

Server log:

[2023-10-21T13:41:01.998Z TRACE rustls::server::hs] we got a clienthello ClientHelloPayload { client_version: TLSv1_2, random: 23364b945e31adda803811e924a3d83bf7115bb8c5d50c6e206b419b6eba15a4, session_id: , cipher_suites: [TLS13_AES_256_GCM_SHA384, TLS13_AES_128_GCM_SHA256, TLS13_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV], compression_methods: [Null], extensions: [SupportedVersions([TLSv1_3]), ECPointFormats([Uncompressed]), NamedGroups([X25519, secp256r1, secp384r1]), SignatureAlgorithms([ECDSA_NISTP384_SHA384, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA512, RSA_PSS_SHA384, RSA_PSS_SHA256, RSA_PKCS1_SHA512, RSA_PKCS1_SHA384, RSA_PKCS1_SHA256]), ExtendedMasterSecretRequest, CertificateStatusRequest(OCSP(OCSPCertificateStatusRequest { responder_ids: [], extensions:  })), SignedCertificateTimestampRequest, KeyShare([KeyShareEntry { group: X25519, payload: ef69ae8dd9d10374247200427a2931ec5e9f6312f1afaf81a4f4c1140cc9ff0a }]), PresharedKeyModes([PSK_DHE_KE]), TransportParameters([1, 2, 103, 16, 3, 2, 69, 192, 4, 8, 255, 255, 255, 255, 255, 255, 255, 255, 5, 4, 128, 19, 18, 208, 6, 4, 128, 19, 18, 208, 7, 4, 128, 19, 18, 208, 8, 2, 64, 100, 9, 2, 64, 100, 14, 1, 5, 64, 182, 0, 32, 4, 128, 0, 255, 255, 15, 8, 1, 84, 78, 242, 68, 246, 41, 232, 106, 178, 0, 192, 0, 0, 0, 255, 4, 222, 26, 2, 67, 232]), SessionTicket(Request)] }
[2023-10-21T13:41:01.999Z TRACE rustls::server::server_conn] sni None
[2023-10-21T13:41:01.999Z TRACE rustls::server::server_conn] sig schemes [ECDSA_NISTP384_SHA384, ECDSA_NISTP256_SHA256, ED25519, RSA_PSS_SHA512, RSA_PSS_SHA384, RSA_PSS_SHA256, RSA_PKCS1_SHA512, RSA_PKCS1_SHA384, RSA_PKCS1_SHA256]
[2023-10-21T13:41:01.999Z TRACE rustls::server::server_conn] alpn protocols None
[2023-10-21T13:41:01.999Z TRACE rustls::server::server_conn] cipher suites [TLS13_AES_256_GCM_SHA384, TLS13_AES_128_GCM_SHA256, TLS13_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
[2023-10-21T13:41:01.999Z DEBUG rustls::server::hs] decided upon suite TLS13_AES_256_GCM_SHA384
[2023-10-21T13:41:01.999Z TRACE rustls::server::tls13::client_hello] sending server hello Message { version: TLSv1_2, payload: Handshake { parsed: HandshakeMessagePayload { typ: ServerHello, payload: ServerHello(ServerHelloPayload { legacy_version: TLSv1_2, random: 3579df1d9768c851fe8d054b69f1310c173fc52969b2a5a3335401be0496b523, session_id: , cipher_suite: TLS13_AES_256_GCM_SHA384, compression_method: Null, extensions: [KeyShare(KeyShareEntry { group: X25519, payload: e267547fe1a5d3e1e61ebee31da5b37d657d52e82590bbd8414f5379da51c430 }), SupportedVersions(TLSv1_3)] }) }, encoded: 0200005603033579df1d9768c851fe8d054b69f1310c173fc52969b2a5a3335401be0496b52300130200002e00330024001d0020e267547fe1a5d3e1e61ebee31da5b37d657d52e82590bbd8414f5379da51c430002b00020304 } }
[2023-10-21T13:41:01.999Z TRACE rustls::server::tls13::client_hello] sending encrypted extensions Message { version: TLSv1_3, payload: Handshake { parsed: HandshakeMessagePayload { typ: EncryptedExtensions, payload: EncryptedExtensions([TransportParameters([1, 2, 103, 16, 3, 2, 69, 192, 4, 8, 255, 255, 255, 255, 255, 255, 255, 255, 5, 4, 128, 19, 18, 208, 6, 4, 128, 19, 18, 208, 7, 4, 128, 19, 18, 208, 8, 2, 64, 100, 9, 2, 64, 100, 14, 1, 5, 64, 182, 0, 2, 16, 48, 73, 210, 6, 76, 208, 179, 123, 150, 175, 204, 81, 253, 9, 165, 40, 32, 4, 128, 0, 255, 255, 0, 20, 204, 251, 150, 183, 155, 15, 114, 129, 192, 229, 132, 236, 140, 253, 107, 198, 174, 150, 124, 56, 15, 8, 44, 153, 198, 202, 75, 234, 83, 25, 16, 8, 11, 203, 71, 225, 153, 247, 194, 129, 106, 178, 0, 192, 0, 0, 0, 255, 4, 222, 26, 2, 67, 232])]) }, encoded: 0800008800860039008201026710030245c00408ffffffffffffffff0504801312d00604801312d00704801312d008024064090240640e010540b60002103049d2064cd0b37b96afcc51fd09a52820048000ffff0014ccfb96b79b0f7281c0e584ec8cfd6bc6ae967c380f082c99c6ca4bea531910080bcb47e199f7c2816ab200c0000000ff04de1a0243e8 } }
[2023-10-21T13:41:01.999Z TRACE rustls::server::tls13::client_hello] sending certificate Message { version: TLSv1_3, payload: Handshake { parsed: HandshakeMessagePayload { typ: Certificate, payload: CertificateTLS13(CertificatePayloadTLS13 { context: , entries: [CertificateEntry { cert: Certificate(b"0\x82\x03\x990\x82\x02\x81\xa0\x03\x02\x01\x02\x02\x14>w)v{$rz\xab\x8eA\xfb\xdbH\xfe\x04Q~\xdc\x840\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\0061\x170\x15\x06\x03U\x04\x03\x0c\x0e51.195.104.1901\x0b0\t\x06\x03U\x04\x06\x13\x02AE1\x0e0\x0c\x06\x03U\x04\x07\x0c\x05Dubai0\x1e\x17\r231021090401Z\x17\r331018090401Z0j1\x0b0\t\x06\x03U\x04\x06\x13\x02AE1\x0e0\x0c\x06\x03U\x04\x08\x0c\x05Dubai1\x0e0\x0c\x06\x03U\x04\x07\x0c\x05Dubai1\x100\x0e\x06\x03U\x04\n\x0c\x07Unknown1\x100\x0e\x06\x03U\x04\x0b\x0c\x07Unknown1\x170\x15\x06\x03U\x04\x03\x0c\x0e51.195.104.1900\x82\x01\"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\0\x03\x82\x01\x0f\00\x82\x01\n\x02\x82\x01\x01\0\xd0}/:\xbcQ\xb3\xe4\xf9\xdc\xcd\xa7\xe7\xa7\x8b\"\xe5r\xf9\xe3\x13/\x16\xf1\xa88\xd1,\xf5\x84\xe0A4\xb7\x01\r\xef\xcbo\xde\\\x93\x88\x1f\xe0H\xfe\xbdz<-\xc4\x84d/\xadd\x95*Ii$\x1f`\xbf\xd5\x8aH3$H\x94\xad\xb1@\xdc\xf2\xe2\x81\x9ag\xceP\xac\x7f]\x95\x15l\xe7IlQ:\x81\xdakE\x86B\x1fW\xb9\xef\x02\x14Q\xe3?\xcd\x17mf\xd1\xfc\xb1)\xc1\xbf\xd0i\x90\xd2\x04\x1f\xd6\xbc\xec\xe2`\xd16\x12Z\xe86[gZ\xa6-j\x7fK\xf1z\x13)\xd8\xa1\x9e\x9e\xc7\x97q\xe0\x0b\x91\xa3h\x95\xbe\xec\xa3u\x8d\xb8\x16\"P\xee\xe8\n\xff\x0f8}Y\xe8$\xff\xd5%R\x8f\xee\x1f\xbf\xac\xe4\x88\x958#O=\x04\x0b\xd9\x96\xe6\x9d \xc4\x7f\x19\xc1\xb0h\xc5C\xf2D\xf3\xbf>\xdd\xfd4\x93\x87\xb5\xd0U\x97\xd5\x05\x03\xae\xd4\xcf\x05\xca\xee\xe7\xa7\xa6\xaf\xe8c'-\x15\x85L-@?\xb6\xf2\x1a^\x03\x0e\xaf\x1d\x02\x03\x01\0\x01\xa3k0i0\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\x10{\xd6^8\x10\x82\x1enH\xd8\xc9)\"9`\x1a\xa2\x85\x950\t\x06\x03U\x1d\x13\x04\x020\00\x0b\x06\x03U\x1d\x0f\x04\x04\x03\x02\x04\xf00\x0f\x06\x03U\x1d\x11\x04\x080\x06\x87\x043\xc3h\xbe0\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x14&\xe4\xce\x9d~\xeb\xd3dy2\xfd\xa6\x13q\xc5\x83\xc09\r\xfb0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\0\x03\x82\x01\x01\0kX\x97\xfek\xbd\x98W\xa3\x17\xf3\xbb4\xcfi(E\xc4\x99\x90~Z%n~7\x9f4\xf7\\\x80x\xe5\n\xd5]\xd1\xce\xdfFR\x86 \x92\x7f\xa7q\x94\x92T _\x9a\xe6\xe1F$\x9c/t\xc6HL\xda3\xb6\x1ay\x85k\xe7\x13\xe272\xfc\r=\xa8\xc0\r\x18sw1\xe8{\xb2\xa3\x9a\xa8\xcb\xae\xde\x0b2\x9d\xff\x86\x8b(\xf4\x9e\xc4m\x12Q\xfe\xfe\xf1\x1b\x1c\x81Gd\xa6-\x8c\r=\xab\xb9\xa4\x94/\x8a,-\xba\xda(\x97\xaa2\x16R\xde\xac\xb7\x99=`\x06N\xaf\xd0+'B6\x1e\xf7\xa5\xcd\xc4\x1a\n\xee\xec8\xb9\xf3\xcd{A>\xf4 \x85\x9c\xbeV\x91\xd9^\x8c\xf73DP\x05\xf0\x86\x17\x80.\xa3d\x9fQ\x85\xbfCA\xdd\xbd\xe7\xa5=y\xca!\xae\xaf_\xbdYT#!\x7f\x94\xdb\xf8&\x0e\xf6m\xd7\xc4\x7f\r\xc9\x9a\x8cs\x8a/\xe0~\x05:\xf4\\Y\x02og\x15\x9c\x17\xb6/\x80{\x9f\xd8=\xc4\xd2\xefN\xb7\xceq\xfc"), exts: [] }, CertificateEntry { cert: Certificate(b"0\x82\x03K0\x82\x023\xa0\x03\x02\x01\x02\x02\x14t_{\xeb\x934\xe8\x93\xfdK\x8c\xdbY\xa9\x82P\x8e\xdc\xeeU0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\0051\x160\x14\x06\x03U\x04\x03\x0c\r192.168.70.651\x0b0\t\x06\x03U\x04\x06\x13\x02AE1\x0e0\x0c\x06\x03U\x04\x07\x0c\x05Dubai0\x1e\x17\r230907190010Z\x17\r330606190010Z051\x160\x14\x06\x03U\x04\x03\x0c\r192.168.70.651\x0b0\t\x06\x03U\x04\x06\x13\x02AE1\x0e0\x0c\x06\x03U\x04\x07\x0c\x05Dubai0\x82\x01\"0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\0\x03\x82\x01\x0f\00\x82\x01\n\x02\x82\x01\x01\0\xa1\xa8\x06/\xf1\xabC\x0b0y\x83}\n5\xd3\xfet\xe9m\x1d\"\xb0\xb2\xf0\xd5%\x90;\x08!I\xb5\xe6<+V\xbd\xd00k\x88\r\r0\xdd\x06\x88\xeeeq\xd4\xde:0\xc0F\x97\xf3\xf9\x9d\xfaj\x92s\xa7!\xd4#T\x87I\xaf\x147\x05&\x8e\xc5 \x837(=iv\xe5\xe74\xbe9O\xf5\xc2\xfe\x8aV\x1d'\xa5\x0c1\xbb\xcc\x02\x9b\xa2\xf6\xf6B>\xfe\x13_\x01\xe2\xfd\x18\x15\x0e\xa6\xa8\xba\xf1\xed\xce\x91\0\x02\n)\x8d\xb8\x7f'f\xd5\x90\x1f\xde\xdb\x1b\x1e\xe3\xdaH\x88\xdf\xce\xac-v\xb5\xfeqH\x97G\xd1\xeb\xe8\xa7\x89\xcbT\x010\xaa\xba\nX(\x18^\xb8\x06v\xad:\xfaF\xfc\xc4\xb4\x88\xdf\xdb\x19A\xf4\xdfLv\xdc\xb7'\x8d\x08xsj%\xa7C\xd7r\xabH\xd1\x1c}S9\xab\x84\x8e\xabm\xcf\x9f\xfa\x18b\x82\xfd\r\x8b]\x04\x95\x8b\xdc,\x15K\x7f\xd7\xb4\xc6\xaa\x14?\xdb\xdb]\xc3\x93\x89\xf0h\x9f\x86\x8cW}\xbe;\x02\x03\x01\0\x01\xa3S0Q0\x1d\x06\x03U\x1d\x0e\x04\x16\x04\x144\x87n/\xfc\xdaK\xff\x9dBT\xc9\xd9hcL\xdb\xa1?\xee0\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x144\x87n/\xfc\xdaK\xff\x9dBT\xc9\xd9hcL\xdb\xa1?\xee0\x0f\x06\x03U\x1d\x13\x01\x01\xff\x04\x050\x03\x01\x01\xff0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\0\x03\x82\x01\x01\0o[\xe7\x90`\x91\xc6\xc7\xf2hP\xd43\xffB\x96:\x1e\xb690\x9e\xf5sU\x87\x9aV\xd8\x87\t\x1cYh\x03H\xe1\x0f2\xe1$4\xde\xfbFw\x0e#\x0b\xe8BS|\x91J\xb9\x15\xf5\x14FH\xce\xbcw\xb1\xae\xb96\xb2\xe6\xf7\x92\x0e\x8a\xb0\xdf\x84\xe4\xa0\xd4\x9a\x19\x8c\x90\xe3\x07\xc2\x8b\xedp\x93\xa9\xf938\xea\"\xf6>3\xf1%\xb0\x92od\x94\xe0\\\xf4^w7\xe8\xdf7\xd0\xbc@\xb7w\xad\xf38\x9e\x1f\xf2\x0e\xa9\xf5\xca\x05\xc1\x93]rB\x12\xd1\xe9a\x9d4\xa8{YT\x06\x8f6^\x1d\xe6\xde\x07t\x13\xf3{\xe54\xa8\x19&\x15\xb6-J;W<\xcav\x8cZ\x10\xa7\xbe.\xfex\xac\xd1\xbb\xe7P7\xa3\xc2V\x87\x91!\xa3\xf5p`l\x97)\x9f\xbcmo\x91\xacS\x04\x06P\xf9\x17\xadr\xfa\xdbMF\x8b&\xe8\x9d\xb1\x15X\xc7\x0cz\x1dQ\xcd\xd4\xdaT\x91\x9at\xdf\xec\0\xfb\xd8\x19\xe2>\xfa\xfdv\x03#\x16\xb3\x02~\xf7\xc3"), exts: [] }] }) }, encoded: 0b0006fa000006f600039d3082039930820281a00302010202143e7729767b24727aab8e41fbdb48fe04517edc84300d06092a864886f70d01010b050030363117301506035504030c0e35312e3139352e3130342e313930310b3009060355040613024145310e300c06035504070c054475626169301e170d3233313032313039303430315a170d3333313031383039303430315a306a310b3009060355040613024145310e300c06035504080c054475626169310e300c06035504070c0544756261693110300e060355040a0c07556e6b6e6f776e3110300e060355040b0c07556e6b6e6f776e3117301506035504030c0e35312e3139352e3130342e31393030820122300d06092a864886f70d01010105000382010f003082010a0282010100d07d2f3abc51b3e4f9dccda7e7a78b22e572f9e3132f16f1a838d12cf584e04134b7010defcb6fde5c93881fe048febd7a3c2dc484642fad64952a4969241f60bfd58a4833244894adb140dcf2e2819a67ce50ac7f5d95156ce7496c513a81da6b4586421f57b9ef021451e33fcd176d66d1fcb129c1bfd06990d2041fd6bcece260d136125ae8365b675aa62d6a7f4bf17a1329d8a19e9ec79771e00b91a36895beeca3758db8162250eee80aff0f387d59e824ffd525528fee1fbface4889538234f3d040bd996e69d20c47f19c1b068c543f244f3bf3eddfd349387b5d05597d50503aed4cf05caeee7a7a6afe863272d15854c2d403fb6f21a5e030eaf1d0203010001a36b3069301f0603551d23041830168014107bd65e3810821e6e48d8c9292239601aa2859530090603551d1304023000300b0603551d0f0404030204f0300f0603551d1104083006870433c368be301d0603551d0e0416041426e4ce9d7eebd3647932fda61371c583c0390dfb300d06092a864886f70d01010b050003820101006b5897fe6bbd9857a317f3bb34cf692845c499907e5a256e7e379f34f75c8078e50ad55dd1cedf46528620927fa771949254205f9ae6e146249c2f74c6484cda33b61a79856be713e23732fc0d3da8c00d18737731e87bb2a39aa8cbaede0b329dff868b28f49ec46d1251fefef11b1c814764a62d8c0d3dabb9a4942f8a2c2dbada2897aa321652deacb7993d60064eafd02b2742361ef7a5cdc41a0aeeec38b9f3cd7b413ef420859cbe5691d95e8cf733445005f08617802ea3649f5185bf4341ddbde7a53d79ca21aeaf5fbd595423217f94dbf8260ef66dd7c47f0dc99a8c738a2fe07e053af45c59026f67159c17b62f807b9fd83dc4d2ef4eb7ce71fc000000034f3082034b30820233a0030201020214745f7beb9334e893fd4b8cdb59a982508edcee55300d06092a864886f70d01010b050030353116301406035504030c0d3139322e3136382e37302e3635310b3009060355040613024145310e300c06035504070c054475626169301e170d3233303930373139303031305a170d3333303630363139303031305a30353116301406035504030c0d3139322e3136382e37302e3635310b3009060355040613024145310e300c06035504070c05447562616930820122300d06092a864886f70d01010105000382010f003082010a0282010100a1a8062ff1ab430b3079837d0a35d3fe74e96d1d22b0b2f0d525903b082149b5e63c2b56bdd0306b880d0d30dd0688ee6571d4de3a30c04697f3f99dfa6a9273a721d423548749af143705268ec5208337283d6976e5e734be394ff5c2fe8a561d27a50c31bbcc029ba2f6f6423efe135f01e2fd18150ea6a8baf1edce9100020a298db87f2766d5901fdedb1b1ee3da4888dfceac2d76b5fe71489747d1ebe8a789cb540130aaba0a5828185eb80676ad3afa46fcc4b488dfdb1941f4df4c76dcb7278d0878736a25a743d772ab48d11c7d5339ab848eab6dcf9ffa186282fd0d8b5d04958bdc2c154b7fd7b4c6aa143fdbdb5dc39389f0689f868c577dbe3b0203010001a3533051301d0603551d0e0416041434876e2ffcda4bff9d4254c9d968634cdba13fee301f0603551d2304183016801434876e2ffcda4bff9d4254c9d968634cdba13fee300f0603551d130101ff040530030101ff300d06092a864886f70d01010b050003820101006f5be7906091c6c7f26850d433ff42963a1eb639309ef57355879a56d887091c59680348e10f32e12434defb46770e230be842537c914ab915f5144648cebc77b1aeb936b2e6f7920e8ab0df84e4a0d49a198c90e307c28bed7093a9f93338ea22f63e33f125b0926f6494e05cf45e7737e8df37d0bc40b777adf3389e1ff20ea9f5ca05c1935d724212d1e9619d34a87b5954068f365e1de6de077413f37be534a8192615b62d4a3b573cca768c5a10a7be2efe78acd1bbe75037a3c256879121a3f570606c97299fbc6d6f91ac53040650f917ad72fadb4d468b26e89db11558c70c7a1d51cdd4da54919a74dfec00fbd819e23efafd76032316b3027ef7c30000 } }
[2023-10-21T13:41:02.002Z TRACE rustls::server::tls13::client_hello] sending certificate-verify Message { version: TLSv1_3, payload: Handshake { parsed: HandshakeMessagePayload { typ: CertificateVerify, payload: CertificateVerify(DigitallySignedStruct { scheme: RSA_PSS_SHA512, sig: aa19a70ac470de59ca5fe2a16602df6bd3da4881cb8baafc7ed58ed1f9d4d0e4ae9c02ebc2b1425afd45d8b853980930b39e305759bccbedd1ce4c9d1b1e8730621164861466c4a7fabaff481ee03a9b7486fec996b7994c90a56b2cd7777c4454aa9556e8833fb89e63bd284b1b2c080ffa6c14fee3178fc93f46241e1631775ebac257bbdbaab1dcd8ee022c40241b743b56b31e41ec2f0ddb850c91828bdbac4b0143cfbf34de604b7462e5110bc3db77df296fa7c3e52284eaa12bb5be78e35a0a8c954db36851d5a93fcff01a511fa13765ef4c436d7d719e2275f8c59cfa585f5782065afd01caf6332ec1a5ba9aceaafc4d61d1514a2d8f34126c717f }) }, encoded: 0f00010408060100aa19a70ac470de59ca5fe2a16602df6bd3da4881cb8baafc7ed58ed1f9d4d0e4ae9c02ebc2b1425afd45d8b853980930b39e305759bccbedd1ce4c9d1b1e8730621164861466c4a7fabaff481ee03a9b7486fec996b7994c90a56b2cd7777c4454aa9556e8833fb89e63bd284b1b2c080ffa6c14fee3178fc93f46241e1631775ebac257bbdbaab1dcd8ee022c40241b743b56b31e41ec2f0ddb850c91828bdbac4b0143cfbf34de604b7462e5110bc3db77df296fa7c3e52284eaa12bb5be78e35a0a8c954db36851d5a93fcff01a511fa13765ef4c436d7d719e2275f8c59cfa585f5782065afd01caf6332ec1a5ba9aceaafc4d61d1514a2d8f34126c717f } }
[2023-10-21T13:41:02.002Z TRACE rustls::server::tls13::client_hello] sending finished Message { version: TLSv1_3, payload: Handshake { parsed: HandshakeMessagePayload { typ: Finished, payload: Finished(c3ecd6a817d2569285fa7ebe1ea73a0bb07f783f2295584b2825abea7ed8589f648f8d768907122fc226ba380ed7bdbf) }, encoded: 14000030c3ecd6a817d2569285fa7ebe1ea73a0bb07f783f2295584b2825abea7ed8589f648f8d768907122fc226ba380ed7bdbf } }
[2023-10-21T13:41:02.002Z TRACE tracing::span::active] <- frame;
[2023-10-21T13:41:02.002Z TRACE tracing::span] -- frame;
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection] Handshake keys ready
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection] wrote 90 Initial CRYPTO bytes
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection] Data keys ready
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection] wrote 2246 Handshake CRYPTO bytes
[2023-10-21T13:41:02.002Z TRACE tracing::span::active] <- first recv;
[2023-10-21T13:41:02.002Z TRACE tracing::span] -- first recv;
[2023-10-21T13:41:02.002Z TRACE quinn_proto::endpoint] connection incoming id=0 icid=0bcb47e199f7c281
[2023-10-21T13:41:02.002Z INFO  thread::eventloop::iouring::uni::quic::quinn::instance] server Accepting NewConnection:ConnectionHandle(0) event from 94.204.197.121:37092
[2023-10-21T13:41:02.002Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll_endpoint_events
[2023-10-21T13:41:02.002Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=0
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection] ACK ArrayRangeSet([1..2]), Delay = 0us
[2023-10-21T13:41:02.002Z TRACE quinn_proto::connection] CRYPTO: off 0 len 90
[2023-10-21T13:41:02.003Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=0
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] CRYPTO: off 0 len 1009
[2023-10-21T13:41:02.003Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=1
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] CRYPTO: off 1009 len 1153
[2023-10-21T13:41:02.003Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=2
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] CRYPTO: off 2162 len 84
[2023-10-21T13:41:02.003Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] sending 2531 bytes in 3 datagrams
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection::packet_builder] send; space=Data pn=0
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] NEW_CONNECTION_ID sequence=1 id=f9989d015afb1a68
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] NEW_CONNECTION_ID sequence=2 id=ac3aced4ac77ff22
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] NEW_CONNECTION_ID sequence=3 id=ad72142aeac4f632
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] NEW_CONNECTION_ID sequence=4 id=219bb2a4f1241c5a
[2023-10-21T13:41:02.003Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.003Z TRACE quinn_proto::connection] sending 138 bytes in 1 datagrams
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::client] * process data(2531bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:2,master:0) | segm_size:Some(1200) | ecn:Some(Ect0)
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::client] * process data(138bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:3,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:2,master:0)
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:2,master:0), res=2531 | is_more: false
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:2,master:0), is_more:false
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:2
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:3,master:0)
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:3,master:0), res=138 | is_more: false
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:3,master:0), is_more:false
[2023-10-21T13:41:02.003Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:3
[2023-10-21T13:41:02.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 52
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection] PTO fired in_flight=2669 count=0 space=Initial
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=1
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection] CRYPTO: off 0 len 90
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection::packet_builder] PADDING * 1063
[2023-10-21T13:41:02.998Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=2
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection] CRYPTO: off 0 len 90
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection::packet_builder] PADDING * 1063
[2023-10-21T13:41:02.998Z TRACE tracing::span] -- send;
[2023-10-21T13:41:02.998Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:5,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:6,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:5,master:0)
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:5,master:0), res=1200 | is_more: false
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:5,master:0), is_more:false
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:5
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:6,master:0)
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:6,master:0), res=1200 | is_more: false
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:6,master:0), is_more:false
[2023-10-21T13:41:02.998Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:6
[2023-10-21T13:41:03.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 53
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection] PTO fired in_flight=5069 count=1 space=Handshake
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=3
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection] CRYPTO: off 0 len 1009
[2023-10-21T13:41:03.997Z TRACE tracing::span] -- send;
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection] sending 1055 bytes in 1 datagrams
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=4
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection] CRYPTO: off 1009 len 1153
[2023-10-21T13:41:03.997Z TRACE tracing::span] -- send;
[2023-10-21T13:41:03.997Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::client] * process data(1055bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:8,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:9,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:8,master:0)
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:8,master:0), res=1055 | is_more: false
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:8,master:0), is_more:false
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:8
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:9,master:0)
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:9,master:0), res=1200 | is_more: false
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:9,master:0), is_more:false
[2023-10-21T13:41:03.997Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:9
[2023-10-21T13:41:04.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 54
[2023-10-21T13:41:05.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 55
[2023-10-21T13:41:06.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 56
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection] PTO fired in_flight=7324 count=2 space=Initial
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=3
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection] CRYPTO: off 0 len 90
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection::packet_builder] PADDING * 1063
[2023-10-21T13:41:06.994Z TRACE tracing::span] -- send;
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=4
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection] CRYPTO: off 0 len 90
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection::packet_builder] PADDING * 1063
[2023-10-21T13:41:06.994Z TRACE tracing::span] -- send;
[2023-10-21T13:41:06.994Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:11,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:12,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:11,master:0)
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:11,master:0), res=1200 | is_more: false
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:11,master:0), is_more:false
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:11
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:12,master:0)
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:12,master:0), res=1200 | is_more: false
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:12,master:0), is_more:false
[2023-10-21T13:41:06.994Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:12
[2023-10-21T13:41:07.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 57
[2023-10-21T13:41:08.989Z TRACE thread::eventloop::iouring::timerfd] Rearming 58
[2023-10-21T13:41:09.990Z TRACE thread::eventloop::iouring::timerfd] Rearming 59
[2023-10-21T13:41:10.990Z TRACE thread::eventloop::iouring::timerfd] Rearming 60
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection] PTO fired in_flight=9724 count=3 space=Handshake
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=5
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection] CRYPTO: off 2162 len 84
[2023-10-21T13:41:11.989Z TRACE tracing::span] -- send;
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection] sending 131 bytes in 1 datagrams
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection::packet_builder] send; space=Handshake pn=6
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection] CRYPTO: off 0 len 1009
[2023-10-21T13:41:11.989Z TRACE tracing::span] -- send;
[2023-10-21T13:41:11.989Z TRACE quinn_proto::connection] sending 1055 bytes in 1 datagrams
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::client] * process data(131bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:14,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::client] * process data(1055bytes) to Some(Some(94.204.197.121:37092)) | Identity(type:ServerUdp,flag:3,slave:15,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:14,master:0)
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:14,master:0), res=131 | is_more: false
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:14,master:0), is_more:false
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:14
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::server] _found endpoint:0 for Identity(type:ServerUdp,flag:4,slave:15,master:0)
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::server::endpoint] UDP incoming_data Identity(type:ServerUdp,flag:4,slave:15,master:0), res=1055 | is_more: false
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:ServerUdp,flag:4,slave:15,master:0), is_more:false
[2023-10-21T13:41:11.989Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:15
[2023-10-21T13:41:11.990Z TRACE thread::eventloop::iouring::timerfd] Rearming 61
[2023-10-21T13:41:11.998Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-21T13:41:11.998Z TRACE quinn_proto::connection] timeout timer=Idle
[2023-10-21T13:41:11.998Z TRACE quinn_proto::connection] connection closed
[2023-10-21T13:41:11.998Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll_endpoint_events
[2023-10-21T13:41:11.998Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-21T13:41:11.998Z WARN  thread::eventloop::iouring::uni::quic::quinn::instance] Server evt: Connection lost: timed out!
[2023-10-21T13:41:11.999Z INFO  server::app] Worker:0 disconnected with reason: `IOError(Custom { kind: TimedOut, error: TimedOut })`!
[2023-10-21T13:41:11.999Z WARN  thread::eventloop::iouring::uni::quic::quinn::instance] Drop QConn:0 | local_uni_chs: 0, local_bidi: 0, remote_uni: 0, remote_bidi: 0
serzhiio commented 11 months ago

I tried different timeouts, but nothing helps

Ralith commented 11 months ago

It looks like the client never receives anything from the server. Check packet captures on both client and server to see what's actually being sent/received at the kernel level.

serzhiio commented 11 months ago

It looks like the client never receives anything from the server. Check packet captures on both client and server to see what's actually being sent/received at the kernel level.

Client is receiveing packets, but i can't udnerstand why i everytime get timeout.

[2023-10-22T10:50:40.122743Z TRACE quinn_proto::connection] got Initial packet (1200 bytes) from 51.195.*.*:54474 using id a8c5d617ece0dc51
[2023-10-22T10:50:40.122754Z TRACE quinn_proto::connection] recv; space=Initial pn=1
[2023-10-22T10:50:40.122760Z TRACE tracing::span::active] -> recv;
[2023-10-22T10:50:40.122765Z TRACE quinn_proto::connection] switching remote CID to 92a93a8383c1908d
[2023-10-22T10:50:40.122775Z TRACE quinn_proto::connection] frame; ty=CRYPTO
[2023-10-22T10:50:40.122782Z TRACE tracing::span::active] -> frame;
[2023-10-22T10:50:40.122791Z TRACE quinn_proto::connection] consumed 90 CRYPTO bytes
[2023-10-22T10:50:40.122829Z TRACE rustls::client::hs] We got ServerHello ServerHelloPayload {
        legacy_version: TLSv1_2,
        random: 43114072f2c21d1002d91e8dea09647c1ec59fca48e3e82fc9f5529cfdb27ed2,
        session_id: ,
        cipher_suite: TLS13_AES_256_GCM_SHA384,
        compression_method: Null,
        extensions: [
            KeyShare(
                KeyShareEntry {
                    group: X25519,
                    payload: b83857115d22dfe16f4cbd0712be5e1fb801b9bc28b6d4d749ba437f0796b976,
                },
            ),
            SupportedVersions(
                TLSv1_3,
            ),
        ],
    }
[2023-10-22T10:50:40.122868Z DEBUG rustls::client::hs] Using ciphersuite TLS13_AES_256_GCM_SHA384
[2023-10-22T10:50:40.122878Z DEBUG rustls::client::tls13] Not resuming
[2023-10-22T10:50:40.122883Z TRACE rustls::client::client_conn] EarlyData rejected
[2023-10-22T10:50:40.122975Z TRACE tracing::span::active] <- frame;
[2023-10-22T10:50:40.122980Z TRACE tracing::span] -- frame;
[2023-10-22T10:50:40.123022Z TRACE quinn_proto::connection] Handshake keys ready
[2023-10-22T10:50:40.123028Z TRACE tracing::span::active] <- recv;
[2023-10-22T10:50:40.123032Z TRACE tracing::span] -- recv;
[2023-10-22T10:50:40.123038Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-22T10:50:40.123048Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=4
[2023-10-22T10:50:40.123055Z TRACE quinn_proto::connection] ACK ArrayRangeSet([1..2]), Delay = 0us
[2023-10-22T10:50:40.123062Z TRACE quinn_proto::connection::packet_builder] PADDING * 1071
[2023-10-22T10:50:40.123069Z TRACE tracing::span] -- send;
[2023-10-22T10:50:40.123075Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-22T10:50:40.123085Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:7,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-22T10:50:40.123133Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:7,master:0), result=1200, is_more=false
[2023-10-22T10:50:40.123140Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:7,master:0), is_more:false
[2023-10-22T10:50:40.123147Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:7
[2023-10-22T10:50:40.841893Z TRACE exchange] timeout: 1
[2023-10-22T10:50:40.841910Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:40.841915Z TRACE thread::eventloop::iouring::timerfd] Rearming 1
[2023-10-22T10:50:41.841948Z TRACE exchange] timeout: 2
[2023-10-22T10:50:41.841976Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:41.841982Z TRACE thread::eventloop::iouring::timerfd] Rearming 2
[2023-10-22T10:50:41.982441Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-22T10:50:41.982457Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-22T10:50:41.982467Z TRACE quinn_proto::connection] PTO fired in_flight=4800 count=1 space=Initial
[2023-10-22T10:50:41.982474Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-22T10:50:41.982486Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=5
[2023-10-22T10:50:41.982493Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-22T10:50:41.982501Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-22T10:50:41.982513Z TRACE tracing::span] -- send;
[2023-10-22T10:50:41.982523Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-22T10:50:41.982537Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=6
[2023-10-22T10:50:41.982543Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-22T10:50:41.982547Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-22T10:50:41.982553Z TRACE tracing::span] -- send;
[2023-10-22T10:50:41.982558Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-22T10:50:41.982563Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:9,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-22T10:50:41.982573Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:10,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-22T10:50:41.982621Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:9,master:0), result=1200, is_more=false
[2023-10-22T10:50:41.982628Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:9,master:0), is_more:false
[2023-10-22T10:50:41.982634Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:9
[2023-10-22T10:50:41.982638Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:10,master:0), result=1200, is_more=false
[2023-10-22T10:50:41.982644Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:10,master:0), is_more:false
[2023-10-22T10:50:41.982648Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:10
[2023-10-22T10:50:42.842033Z TRACE exchange] timeout: 3
[2023-10-22T10:50:42.842056Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:42.842062Z TRACE thread::eventloop::iouring::timerfd] Rearming 3
[2023-10-22T10:50:43.842090Z TRACE exchange] timeout: 4
[2023-10-22T10:50:43.842106Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:43.842111Z TRACE thread::eventloop::iouring::timerfd] Rearming 4
[2023-10-22T10:50:44.842141Z TRACE exchange] timeout: 5
[2023-10-22T10:50:44.842158Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:44.842163Z TRACE thread::eventloop::iouring::timerfd] Rearming 5
[2023-10-22T10:50:45.842188Z TRACE exchange] timeout: 6
[2023-10-22T10:50:45.842202Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:45.842206Z TRACE thread::eventloop::iouring::timerfd] Rearming 6
[2023-10-22T10:50:45.979361Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-22T10:50:45.979378Z TRACE quinn_proto::connection] timeout timer=LossDetection
[2023-10-22T10:50:45.979388Z TRACE quinn_proto::connection] PTO fired in_flight=7200 count=2 space=Initial
[2023-10-22T10:50:45.979395Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-22T10:50:45.979405Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=7
[2023-10-22T10:50:45.979413Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-22T10:50:45.979422Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-22T10:50:45.979440Z TRACE tracing::span] -- send;
[2023-10-22T10:50:45.979457Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-22T10:50:45.979464Z TRACE quinn_proto::connection::packet_builder] send; space=Initial pn=8
[2023-10-22T10:50:45.979469Z TRACE quinn_proto::connection] CRYPTO: off 0 len 267
[2023-10-22T10:50:45.979473Z TRACE quinn_proto::connection::packet_builder] PADDING * 808
[2023-10-22T10:50:45.979479Z TRACE tracing::span] -- send;
[2023-10-22T10:50:45.979484Z TRACE quinn_proto::connection] sending 1200 bytes in 1 datagrams
[2023-10-22T10:50:45.979489Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:12,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-22T10:50:45.979500Z TRACE thread::eventloop::iouring::uni::client] * process data(1200bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:13,master:0) | segm_size:None | ecn:Some(Ect0)
[2023-10-22T10:50:45.979575Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:12,master:0), result=1200, is_more=false
[2023-10-22T10:50:45.979583Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:12,master:0), is_more:false
[2023-10-22T10:50:45.979587Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:12
[2023-10-22T10:50:45.979592Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:13,master:0), result=1200, is_more=false
[2023-10-22T10:50:45.979597Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:13,master:0), is_more:false
[2023-10-22T10:50:45.979602Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:13
[2023-10-22T10:50:46.842240Z TRACE exchange] timeout: 7
[2023-10-22T10:50:46.842256Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:46.842261Z TRACE thread::eventloop::iouring::timerfd] Rearming 7
[2023-10-22T10:50:47.842295Z TRACE exchange] timeout: 8
[2023-10-22T10:50:47.842311Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:47.842316Z TRACE thread::eventloop::iouring::timerfd] Rearming 8
[2023-10-22T10:50:48.842349Z TRACE exchange] timeout: 9
[2023-10-22T10:50:48.842365Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:48.842369Z TRACE thread::eventloop::iouring::timerfd] Rearming 9
[2023-10-22T10:50:49.842415Z TRACE exchange] timeout: 10
[2023-10-22T10:50:49.842449Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:49.842455Z TRACE thread::eventloop::iouring::timerfd] Rearming 10
[2023-10-22T10:50:50.842492Z TRACE exchange] timeout: 11
[2023-10-22T10:50:50.842506Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:50.842512Z TRACE thread::eventloop::iouring::timerfd] Rearming 11
[2023-10-22T10:50:51.842543Z TRACE exchange] timeout: 12
[2023-10-22T10:50:51.842558Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:51.842564Z TRACE thread::eventloop::iouring::timerfd] Rearming 12
[2023-10-22T10:50:51.983056Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - handle timeout
[2023-10-22T10:50:51.983075Z TRACE quinn_proto::connection] timeout timer=Idle
[2023-10-22T10:50:51.983082Z TRACE quinn_proto::connection] connection closed
[2023-10-22T10:50:51.983090Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll_endpoint_events
[2023-10-22T10:50:51.983095Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] - - poll + poll_transmit | max_datagrams_gso:64
[2023-10-22T10:50:51.983102Z WARN  thread::eventloop::iouring::uni::quic::quinn::instance] Client evt: Connection lost: timed out!
[2023-10-22T10:50:51.984358Z TRACE mpub::app] Server Disconnected! IOError(Custom { kind: TimedOut, error: TimedOut }) | Some(0)
[2023-10-22T10:50:51.984368Z WARN  thread::eventloop::iouring::uni::quic::quinn::instance] Drop QConn:0 | local_uni_chs: 0, local_bidi: 0, remote_uni: 0, remote_bidi: 0
[2023-10-22T10:50:52.842600Z TRACE exchange] timeout: 13
[2023-10-22T10:50:52.842615Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:52.842621Z TRACE thread::eventloop::iouring::timerfd] Rearming 13
[2023-10-22T10:50:53.842651Z TRACE exchange] timeout: 14
[2023-10-22T10:50:53.842664Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:53.842668Z TRACE thread::eventloop::iouring::timerfd] Rearming 14
[2023-10-22T10:50:54.842702Z TRACE exchange] timeout: 15
[2023-10-22T10:50:54.842718Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:54.842724Z TRACE thread::eventloop::iouring::timerfd] Rearming 15
[2023-10-22T10:50:55.558483Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:0,slave:0,master:0), result=269, is_more=true
[2023-10-22T10:50:55.558498Z TRACE thread::eventloop::iouring::uni::client] RcvdMsgHdrStorage read_out(data=269bytes)
[2023-10-22T10:50:55.558501Z TRACE thread::eventloop::iouring::uni::client] RecvMeta { len: 133, stride: 133, ecn: Some(Ect0), dst_ip: Some(192.168.70.65) }
[2023-10-22T10:50:55.558505Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] client incoming data to read len=133bytes from 51.195.*.*:54474
[2023-10-22T10:50:55.558511Z DEBUG quinn_proto::endpoint] packet for unrecognized connection a8c5d617ece0dc51
[2023-10-22T10:50:55.558517Z DEBUG quinn_proto::endpoint] sending stateless reset for a8c5d617ece0dc51 to 51.195.*.*:54474
[2023-10-22T10:50:55.558523Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] client Response
[2023-10-22T10:50:55.558527Z TRACE thread::eventloop::iouring::uni::client] * process data(123bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:15,master:0) | segm_size:None | ecn:None
[2023-10-22T10:50:55.558550Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:15,master:0), result=123, is_more=false
[2023-10-22T10:50:55.558559Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:15,master:0), is_more:false
[2023-10-22T10:50:55.558561Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:15
[2023-10-22T10:50:55.842763Z TRACE exchange] timeout: 16
[2023-10-22T10:50:55.842779Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:55.842783Z TRACE thread::eventloop::iouring::timerfd] Rearming 16
[2023-10-22T10:50:56.842824Z TRACE exchange] timeout: 17
[2023-10-22T10:50:56.842839Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:56.842845Z TRACE thread::eventloop::iouring::timerfd] Rearming 17
[2023-10-22T10:50:57.842884Z TRACE exchange] timeout: 18
[2023-10-22T10:50:57.842900Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:57.842906Z TRACE thread::eventloop::iouring::timerfd] Rearming 18
[2023-10-22T10:50:58.842947Z TRACE exchange] timeout: 19
[2023-10-22T10:50:58.842965Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:58.842970Z TRACE thread::eventloop::iouring::timerfd] Rearming 19
[2023-10-22T10:50:59.843001Z TRACE exchange] timeout: 20
[2023-10-22T10:50:59.843015Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:50:59.843021Z TRACE thread::eventloop::iouring::timerfd] Rearming 20
[2023-10-22T10:51:00.843071Z TRACE exchange] timeout: 21
[2023-10-22T10:51:00.843096Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:51:00.843101Z TRACE thread::eventloop::iouring::timerfd] Rearming 21
[2023-10-22T10:51:01.693445Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:0,slave:0,master:0), result=234, is_more=true
[2023-10-22T10:51:01.693464Z TRACE thread::eventloop::iouring::uni::client] RcvdMsgHdrStorage read_out(data=234bytes)
[2023-10-22T10:51:01.693470Z TRACE thread::eventloop::iouring::uni::client] RecvMeta { len: 98, stride: 98, ecn: None, dst_ip: Some(192.168.70.65) }
[2023-10-22T10:51:01.693476Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] client incoming data to read len=98bytes from 51.195.*.*:54474
[2023-10-22T10:51:01.693483Z DEBUG quinn_proto::endpoint] packet for unrecognized connection 6ad7ee0f5199b9f7
[2023-10-22T10:51:01.693491Z DEBUG quinn_proto::endpoint] sending stateless reset for 6ad7ee0f5199b9f7 to 51.195.*.*:54474
[2023-10-22T10:51:01.693500Z TRACE thread::eventloop::iouring::uni::quic::quinn::instance] client Response
[2023-10-22T10:51:01.693505Z TRACE thread::eventloop::iouring::uni::client] * process data(96bytes) to Some(Some(51.195.*.*:54474)) | Identity(type:Client,flag:3,slave:17,master:0) | segm_size:None | ecn:None
[2023-10-22T10:51:01.693535Z TRACE thread::eventloop::iouring::client] incoming DATA Identity(type:Client,flag:4,slave:17,master:0), result=96, is_more=false
[2023-10-22T10:51:01.693541Z TRACE thread::eventloop::iouring::uni::client] notification Send(Msg)Zc flag, Identity(type:Client,flag:4,slave:17,master:0), is_more:false
[2023-10-22T10:51:01.693547Z TRACE thread::eventloop::iouring::uni::client] Removed buffer for id:17
[2023-10-22T10:51:01.843128Z TRACE exchange] timeout: 22
[2023-10-22T10:51:01.843148Z TRACE commonlib::helpers::http_connections] maintain pool with cap:0/1
[2023-10-22T10:51:01.843152Z TRACE thread::eventloop::iouring::timerfd] Rearming 22
Ralith commented 11 months ago

It's timing out because it's not receiving any packets after the first. You should investigate why.

serzhiio commented 11 months ago

I am behind NAT of my provider. Is there a way to uderstand what is happening? client tcpdump

tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
22:53:30.535228 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:30.665646 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 144)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 116
22:53:30.665878 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:31.665443 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:31.665455 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:31.795876 eno1  In  IP (tos 0x2,ECT(0), ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 1200
22:53:31.796265 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:33.663549 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:33.663560 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:37.659723 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:37.659745 eno1  Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 1200
22:53:47.557642 eno1  In  IP (tos 0x2,ECT(0), ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 161)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 133
22:53:47.557735 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 116)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 88
22:53:53.685669 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 98)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 70
22:53:53.685827 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 87)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 59
22:53:54.686122 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 83)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 55
22:53:54.686254 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 77)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 49
22:53:55.557934 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 70)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 42
22:53:55.558021 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 41
22:53:56.557864 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 68)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 40
22:53:56.557988 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 67)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 39
22:53:57.557849 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 66)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 38
22:53:57.557960 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 65)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 37
22:53:57.687681 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 64)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 36
22:53:57.687833 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 63)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 35
22:53:58.557431 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 62)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 34
22:53:58.557555 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 61)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 33
22:53:59.557482 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 60)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 32
22:53:59.557633 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 59)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 31
22:53:59.686817 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 58)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 30
22:53:59.686957 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 57)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 29
22:54:00.557429 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 56)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 28
22:54:00.557618 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 55)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 27
22:54:01.557751 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 54)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 26
22:54:01.557879 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 53)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 25
22:54:01.687019 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 52)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 24
22:54:01.687172 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 51)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 23
22:54:02.558346 eno1  In  IP (tos 0x0, ttl 51, id 0, offset 0, flags [DF], proto UDP (17), length 50)
    51.195.*.*.54474 > 192.168.70.65.50364: UDP, length 22
22:54:02.558480 eno1  Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 49)
    192.168.70.65.50364 > 51.195.*.*.54474: UDP, length 21

Server tcpdump

tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
18:53:30.601596 enp3s0f0 In  IP (tos 0x2,ECT(0), ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 1200
18:53:30.601870 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 144)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 116
18:53:30.732600 enp3s0f0 In  IP (tos 0x2,ECT(0), ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 1200
18:53:30.737484 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 2561)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 2533
18:53:30.737488 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 166)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 138
18:53:31.732311 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1200
18:53:31.732316 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1200
18:53:32.731871 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1083)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1055
18:53:32.731888 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1200
18:53:35.729101 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1200
18:53:35.729115 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1228)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1200
18:53:40.724884 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 161)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 133
18:53:40.724899 enp3s0f0 Out IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 1083)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 1055
18:53:53.621966 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 116)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 88
18:53:53.622237 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 98)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 70
18:53:54.622235 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 87)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 59
18:53:54.622502 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 83)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 55
18:53:54.752490 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 77)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 49
18:53:54.752719 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 70)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 42
18:53:55.624613 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 69)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 41
18:53:55.624861 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 68)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 40
18:53:56.624548 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 67)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 39
18:53:56.624821 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 66)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 38
18:53:57.624403 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 65)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 37
18:53:57.624616 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 64)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 36
18:53:57.753587 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 63)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 35
18:53:57.753774 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 62)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 34
18:53:58.623661 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 61)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 33
18:53:58.623854 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 60)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 32
18:53:59.623481 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 59)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 31
18:53:59.623677 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 58)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 30
18:53:59.753448 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 57)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 29
18:53:59.753614 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 56)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 28
18:54:00.623539 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 55)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 27
18:54:00.623730 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 54)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 26
18:54:01.623529 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 53)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 25
18:54:01.623726 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 52)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 24
18:54:01.753473 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 51)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 23
18:54:01.753663 enp3s0f0 Out IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 50)
    51.195.*.*.54474 > 94.204.197.121.50364: UDP, length 22
18:54:02.624397 enp3s0f0 In  IP (tos 0x0, ttl 50, id 0, offset 0, flags [DF], proto UDP (17), length 49)
    94.204.197.121.50364 > 51.195.*.*.54474: UDP, length 21
Ralith commented 11 months ago

An NAT is probably not relevant. It looks like the server is indeed sending many more packets than are being received. Perhaps GSO is broken in your cloud environment? Have you tried turning that off?

serzhiio commented 11 months ago

An NAT is probably not relevant. It looks like the server is indeed sending many more packets than are being received. Perhaps GSO is broken in your cloud environment? Have you tried turning that off?

Do you mean turn off on network level or just in code?

Ralith commented 11 months ago

No harm in trying both.

serzhiio commented 11 months ago

How to turn it off in code? Just set max_gso_segments to 1?

serzhiio commented 11 months ago

I moved client to AWS server and tried to connect it to my Germany OVH server and voila, it is working. It looks like in Dubai they are blocking this kind of traffic or something. At least all messengers like Whatsup and Telegram have no ability to accept calls.

Ralith commented 11 months ago

How to turn it off in code? Just set max_gso_segments to 1?

TransportConfig::enable_segmentation_offload was added semi-recently in git.

It looks like in Dubai they are blocking this kind of traffic or something

Interesting! Might be informative to experiment with what UDP traffic does and doesn't make it through. In the mean time, certainly makes sense to take your business to a provider that carries your traffic.

serzhiio commented 11 months ago

I need a fresh mind. I have another but a bit similar trouble. I turn on client first time and everything is Ok, then after some working time, disconnect happen and client try to reconnect and everytime it shows TImedOut error, but when i restart client it connects as supposed, then everything repeats...

Ralith commented 11 months ago

Have you configured an appropriate keep-alive interval?

serzhiio commented 11 months ago

I use 10secs idle and 5sec keep-alive.

serzhiio commented 11 months ago

Okay, now the same problem started :(

Ralith commented 11 months ago

Does it still occur when GSO is disabled?

If you can prove that UDP packets are being transmitted from the server and not received by your client, you should work with your cloud provider and/or client-side ISP to find out what's happening to them.

serzhiio commented 11 months ago

GSO makes no sense, tried everything (incl. transport.enable_segmentation_offload(false);).

I am running both client and server and everything is fine, then i kill client and restart everything fine again, then i kill server and restart client sends Hello, server sees it and TimedOut error. If i kill+restart client it connects again as supposed.

serzhiio commented 11 months ago

Looks like the problem was in OVH firewall.

serzhiio commented 11 months ago

Thank you very much for helping!

Ralith commented 11 months ago

Glad you were able to track the issue down!