Steps to reproduce

1. Use quinn to establish quic connection to remote server in context of iOS application via code like this (only UDP socket configuration provided):

   let bind_addr = if addr.is_ipv4() {
   SocketAddr::new(IpAddr::V4(Ipv4Addr::UNSPECIFIED), 0)
   } else {
   SocketAddr::new(IpAddr::V6(Ipv6Addr::UNSPECIFIED), 0)
   };
   let socket = tokio::net::UdpSocket::bind(bind_addr).await?;
   let local_addr = socket.local_addr()?;
   let runtime = quinn::TokioRuntime;
   let endpoint = quinn::Endpoint::new(
   quinn_endpoint_config,
   None,
   socket.into_std()?,
   Arc::new(runtime),
   )?;

2. Minimize iOS application and lock the phone screen while previously established quic connection was alive;
3. Wait several minutes in background (not sure about exact timing, maybe only important to app being actually suspended by iOS);
4. Resume iOS application and try to use previous quic connection such that quinn send some packets.

Actual result

1. quinn is unable to send any UDP packet over previously constructed UDP socket. It receives "Broken Pipe" error from sendmsg; Non of the IP packets leave the device;
2. Idle timer on local side is reset before sendmsg is called and failed, prolonging automatic detection of socket broken;
3. quinn connection is closed by local side once idle timeout is reached.

Expected result

• There is some quick way to know that underlying socket is broken or whole quinn connection is not usable anymore.
• Maybe there is an kind of event that UDP socket rotation is required, so user code can listen/wait to it and call Endpoint::rebind once problem with socket detected.
• Maybe additional API from Connection is exposed, like explicit ping method which can return io::Error from underlying UDP socket and user code can trigger rebind or complete Endpoint/Connection.
• Idle timer preferably should not restart when sendmsg call failed.

More details

I've tracked down EPIPE error from sendmsg inside XNU kernel. The backtrace to likely origin from XNU source code:

• sosendcheck
• sosend
• sendit
• sendmsg_nocancel
• sendmsg

It look like the socket state contains SOF_DEFUNCT or SS_CANTSENDMORE so it's practically become unusable.

According to FreeBSD documentation EPIPE is also indicator of "you can not send more data via provided socket"

See: https://man.freebsd.org/cgi/man.cgi?send(2)

[EPIPE] The socket is unable to send anymore data (SBS_CANTSENDMORE has been set on the socket). This typically means that the socket is not connected.

UPD: The same "broken" state of socket produces ENOTCONN error when called with recvmsg, the origin of it seems to be this check in function soreceive.

UPD 2: Seems ENOTCONN should also have some special treatment on Darwin https://github.com/libevent/libevent/pull/1031/files

cc @mxinden might be of interest.

Many UDP sendmsg failures are transient (e.g. due to wifi dropping out, switching networks, or even spoofed ICMP messages), so we don't generally want to act on them too eagerly, but perhaps this case merits special treatment.

Maybe there is an kind of event that UDP socket rotation is required

While technically feasible, this seems tricky from an ergonomics standpoint. Users should be able to get reasonable behavior by default without having to know about this. Would it make sense for us to try to create a new socket at the same address whenever we get an error like that? How do we square that with user-provided sockets that might have other attached state we don't know about?

What happens to a TCP connection on iOS under the same circumstances?

Maybe additional API from Connection is exposed, like explicit ping method which can return io::Error from underlying UDP socket

I think this is likely to lead to people reimplementing the existing idle timeout mechanism, or inadvertently degrading it. We should try to handle this type of issue automatically.

Idle timer preferably should not restart when sendmsg call failed.

This is an interesting point. Even for transient errors, it's not constructive to sit around waiting for a response that we could've inferred won't happen. We'll need to tweak quinn-proto's API to support this, though. Maybe a Connection::handle_transmit_success method, or similar?

What happens to a TCP connection on iOS under the same circumstances?

I believe every connection is in a broken state after app was in background and they are recreated. Here is how it looks from logs provided by system frameworks:

Maybe there is an kind of event that UDP socket rotation is required

While technically feasible, this seems tricky from an ergonomics standpoint. Users should be able to get reasonable behavior by default without having to know about this. Would it make sense for us to try to create a new socket at the same address whenever we get an error like that? How do we square that with user-provided sockets that might have other attached state we don't know about?

Would be interesting, too, to understand what happens on Android in these kinds of scenarios.

As for "should be able to get reasonable behavior by default" it seems like the current behavior works well outside of mobile, so maybe it we should have some kind of wrapper AsyncUdpSocketBuilder trait that, in the default configuration, doesn't really do anything other than wrapping an existing Arc<dyn AsyncUdpSocket> but implementations of which can do the stuff we need for mobile environments?

I expect to investigate Android next week and report findings here. So far our QA is reported that connection is also does not work after BG until long timeout expires (probably idle timeout). But I guess Android has problem due to a different technical reason. Speculatively I can guess that Android might have more tricker problem which is timers are not advanced in background (https://github.com/tokio-rs/tokio/issues/3185, https://github.com/rust-lang/rust/issues/71860), but as I've said I didn't have a chance to debug problem myself and will likely to do it next week.

@mstyura who's we, by the way? Always good to understand our downstream consumer's use cases.

I believe every connection is in a broken state after app was in background and they are recreated.

Are they recreated automatically, or is it the application's responsibility?

It's interesting to note that Apple explicitly instructs application developers to release resources upon entering the background. Is it typical for libraries to handle that internally? It might be most idiomatic for your application layer to manually tear down/rebuild networking resources from the top level.

It's very much depends of type of connection. According to observed logs connections maintained under the hood by URLSession (apple's http request api) are failed and re-created automatically by the system framework providing http api. The manually created connections like NWConnection are getting event that they are on failed state after BG and it's app responsibility to recover them.

Regarding releasing resources in background, I believe Apple recommendations are mostly about different story - their main concern is releasing memory when app is not visible to user, so more apps can be preserved in memory at same time allowing faster switch between them. It's common for app on iOS to have some network communications after it's already minimized and you cab handle that, but can not handle the point when OS stop executing any thread of the app.

Whether or not library handles socket disruption automatically is very much dependent on library. E.g for http client library like system provided URLSession it's natural to expect it handles reconnection automatically. For more low level sockets when they are initiated by user of the library I expect some kind of event when socket become broken and it's library user responsibility to handle recovery.

Even now I believe Quinn is extensible enough such that I can provide custom AsyncUdpSocket which monitor EPIPE and signal back to my code to handle recovery. So far I've managed to handle stuck connection by re-creating them on EPIPE, which sound like ok but not perfect. It feels like better recovery could be to try rebind_abstract to make quic try to resume existing connection, but I've not yet completed with implementation of this to see if it actually work like this.

Even if quinn will never by default provide any kind of signal to recover broken udp socket I would expect by default it should have been failed faster in such case e.g. by emitting internal event on I/O error from Connection driver to cause connection be LocalClosed with some error message. In such case whoever need different behavior could still provide custom socket implementation.

connections maintained under the hood by URLSession (apple's http request api) are failed and re-created automatically by the system framework providing http api. The manually created connections like NWConnection are getting event that they are on failed state after BG and it's app responsibility to recover them.

Thanks, that's useful context.

Because we're (presumably?) not in a good position to detect when the connection can be resumed, and rebuilding UDP sockets internally is tricky, I see two reasonable paths forwards:

• Fail the connection outright with a (new?) ConnectionError that might prompt the application to reconnect. This is minimal and simple to implement.
• Introduce a new non-fatal event that the application can act on to rebind. This is a larger change, but might save reconnection costs and be easier to integrate into application logic, since Quinn objects won't need to be replaced.

Fail the connection outright with a (new?) ConnectionError that might prompt the application to reconnect. This is minimal and simple to implement.

In one of the attempts to find workaround I've tried to return Err from sendmsg to fail fast connection, but what I've actually got is stuck connection. So maybe the event should be triggered from the connection driver completed with error as well? https://github.com/quinn-rs/quinn/blob/41850c8a304f09c7d009a6e70e48f35bd737e1b5/quinn/src/connection.rs#L67

So maybe the event should be triggered from the connection driver completed with error

Currently the assumption is that the connection driver should never fail. Revisiting that does seem like a reasonable implementation strategy for this new class of errors. It's also probably a bad thing that the I/O driver failing causes connections to get stuck indefinitely regardless.

On Android the situation with sockets in background is occurred to be unrelated to timers. Whenever device screen is locked while app was in foreground the next attempt to call sendmsg completes with EACCESS. quinn tries to send more packets including re-transmission and subsequent sendmsg are not happening probably because according to socket status it's not writable and tokio awaiting it become writeable. Eventually quinn decides that network congestion happen and block sending more packets. After app come to foreground socket become writable again, but probably estimation of available bandwidth was "ruined" by packets which were scheduled to sent but not actually left device, so it take more time to connection to recover from this situation.