olix0r
commented
6 years ago probably. introduced in 71370ac06aad032572e22fdfc1398666d874fdfd. @seanmonstar, do you mind taking a look into this?
Closed siggy closed 6 years ago
olix0r
commented
6 years ago probably. introduced in 71370ac06aad032572e22fdfc1398666d874fdfd. @seanmonstar, do you mind taking a look into this?
hawkw
commented
6 years ago IIRC this test was "fixed" once already in 7f54b5253ded69a5ba599efe135afe5851099e65...evidentially there's more going on here.
seanmonstar
commented
6 years ago I've been looking at this, and so far cannot determine why this error is happening. I don't know if it's flakiness in the test, but I'm a little worried that it's actually flakiness in our TCP proxy... :cry:
The logs here, combined with my commentary on how the code works, shows this:
assert_eq!(&read[..n], expected_msg) is clearly not true ("" != "blah blah").expected_msg2, and then it sees the connection aborted, and the unit test just completely fails.The read of 0 bytes in (3) worries me. That means the proxy closed (or SHUT_WR at least) the connection to the TCP server. I don't know why it would have done so if both sides hadn't closed yet.
Reproducing this is not easy, so trying to capture some logs is 💀 .
olix0r
commented
6 years ago Just hit this again in CI...
---- inbound_tcp stdout ----
thread 'inbound_tcp' panicked at 'read: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', libcore/result.rs:945:5
failures:
inbound_tcp
test result: FAILED. 20 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
error: test failed, to rerun pass '--test transparency'perhaps the CI networking stack is... very unreliable?
olix0r
commented
6 years ago similar but different
---- tcp_waits_for_proxies_to_close stdout ----
thread 'tcp_waits_for_proxies_to_close' panicked at 'read: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.---- inbound_tcp stdout ----
tcp server (addr=127.0.0.1:40015): running
proxy running; control=127.0.0.1:45837, inbound=127.0.0.1:45659 (SO_ORIGINAL_DST=127.0.0.1:40015), outbound=127.0.0.1:40987, metrics=127.0.0.1:40681
tcp client (addr=127.0.0.1:45659) thread running
tcp client (addr=127.0.0.1:45659): connected
tcp client (addr=127.0.0.1:45659): write
tcp client (addr=127.0.0.1:45659): read
thread 'inbound_tcp' panicked at 'TcpConn(addr=127.0.0.1:45659) read() error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }', proxy/tests/support/tcp.rs:109:17
server Listening dropped; addr=127.0.0.1:40015
failures:
inbound_tcpJust saw inbound_tcp flake out on Travis again. Test output looks like it's pretty much the same as what @olix0r posted above but I figured I'd post it here just in case.
hawkw
commented
6 years ago Just saw a failure of tcp_connections_close_if_client_closes which also seems spurious (on a build of PR #865):
seanmonstar
commented
6 years ago I have been pondering setting our Rust test concurrency to 1... The suite would take slightly longer to complete, but we don't have that many tests...
siggy
commented
6 years ago Closing due to inactivity.
master ci failing with
tcp_waits_for_proxies_to_close, from https://travis-ci.org/runconduit/conduit/jobs/367436260#L1: