Reliable way to detect if emulator is up

[matejcik]

Currently the library repeatedly sends UDP message PINGPING, until the emulator responds with PONGPONG. If that does not happen within 1 second, emulator is considered dead.

This opens some interesting race conditions. In particular, what occasionally seems to happen in test runs:

• host sends PINGPING and does not get a reply within the short waiting interval
• host sends PINGPING again
• emulator responds PONGPONG to first PINGPING
• host sends protobuf message and waits for reply
• emulator responds PONGPONG to second PINGPING
• host receives PONGPONG instead of expected protobuf

The reason host hammers emulator with PINPINGs is that as of workflow-restarts, there's no guarantee that a running emulator will respond to a PONGPONG in a timely fashion.

There are various possible solutions to the issue. We need to figure out one that's actually reliable and not affected by whether, e.g., the emulator is currently running an event loop.

[matejcik]

FWIW: something wrong is going on with socket.settimeout in udp.py. When I'm very insistent in code that a timeout of 10 seconds should be used when waiting for the PONGPONG response, it works nicely without hammering. So that might be a path towards a quick fix.

More generally, however, it's still not doing it reliably.