Jepsen: NoHostAvailableException with a list of hosts tried should be treated as info/unknown instead of fail

ttyusupov commented 5 years ago

Currently, CQL client in YugaByte Jepsen tests treats NoHostAvailableException as definitely failed. But it Java driver can throw this exception with a list of hosts tried (for example: NoHostAvailableException: All host(s) tried for query failed (tried: n2/192.168.122.12:9042 (com.datastax.driver.core.exceptions.TransportException: [n2/192.168.122.12:9042] Connection has been closed)) in following scenario:

The client sends write to YugaByte CQL Proxy (node n2).
YugaByte CQL proxy (n2) sends write RPC to the tablet server leader (node n4).
Tablet server hosting CQL Proxy (n2) quickly got killed (in this case by testing framework) right after sending write RPC.
Tablet server on n4 successfully processes write and update DB state.

That means NoHostAvailableException with a list of hosts should be treated as :info (unknown) instead of :fail (definitely failed). We should only treat NoHostAvailableException as :fail when it has no host was tried in error message.

aphyr commented 5 years ago

Good catch, thanks!

ttyusupov commented 5 years ago

Commit: https://github.com/YugaByte/jepsen/commit/ed958c59015cf0660d38547dc7d7a3acd3041d7c

yugabyte / yugabyte-db

Jepsen: NoHostAvailableException with a list of hosts tried should be treated as info/unknown instead of fail #846