Major unstability when waiting for the API socket to be available at VM level

[kanpov]

calledResult::unwrap()on anErrvalue: ApiError(ConnectionError(ApiRequestError(hyper_util::client::legacy::Error(Connect, Os { code: 111, kind: ConnectionRefused, message: "Connection refused" }))))

The current implementation of ensuring the socket file exists clearly has an unstability problem. The issue is severe and may warrant a backport to 0.5.

Potential solutions:

• Try to use inotify for polling?
• Poll whether the socket is connectable?
• Make a connection pool lazily and retry a "/" GET request until it works

Generally, polling for this connectivity is quite difficult and is not made any easier by firecracker not exposing basically any means of waiting for the VM to start up.

[kanpov]

Inotify maybe is the "proper" solution since it gives out fds which are extremely compatible with async, reading from the fds though is a whole another issue that might necessitate pulling in unwanted deps.

[kanpov]

Doing inotify either via libc or inotify-sys is trivial, but this portion might need to be vendored in partly: https://github.com/hannobraun/inotify-rs/blob/main/src/stream.rs