Some tests are unstable

[CharileBrown]

Unstable tests mean they sometimes pass and some other times fail,and some tests may not be run all the time.

1 packetdrill/gtests/net/tcp/blocking_blocking-accept_ipv4. is unstable. 2 packetdrill/gtests/net/tcp/blocking_blocking-write_ipv4-mapped-v6. is unstable. 3 packetdrill/gtests/net/tcp/cwnd_moderation_cwnd-moderation-disorder-no-moderation_ipv4-mapped-v6. is unstable. 4 packetdrill/gtests/net/tcp/shutdown_shutdown-rd-wr-close_ipv6. is unstable. 5 packetdrill/gtests/net/tcp/blocking_blocking-accept_ipv4-mapped-v6. is unstable. 6 packetdrill/gtests/net/tcp/blocking_blocking-connect_ipv6. is unstable. 7 packetdrill/gtests/net/tcp/epoll_epoll_in_edge_ipv6. is unstable. 8 packetdrill/gtests/net/tcp/shutdown_shutdown-rdwr-write-queue-close_ipv4. is unstable. 9 packetdrill/gtests/net/tcp/blocking_blocking-connect_ipv4. is unstable. 10 packetdrill/gtests/net/tcp/tcp_info_tcp-info-sndbuf-limited_ipv4-mapped-v6. is unstable. 11 packetdrill/gtests/net/tcp/timestamping_partial_ipv4-mapped-v6. is unstable. 12 packetdrill/gtests/net/tcp/sack_sack-shift-sacked-7-5-6-8-9-fack_ipv4. is unstable. 13 packetdrill/gtests/net/tcp/ts_recent_reset_tsval_ipv4-mapped-v6. is unstable. 14 packetdrill/gtests/net/tcp/user_timeout_user-timeout-probe_ipv6. is unstable. 15 packetdrill/gtests/net/tcp/timestamping_server_ipv4-mapped-v6. is unstable. 16 packetdrill/gtests/net/tcp/zerocopy_basic_ipv6. is unstable. 17 packetdrill/gtests/net/tcp/zerocopy_closed_ipv6. is unstable. 18 packetdrill/gtests/net/tcp/zerocopy_epoll_oneshot_ipv6. is unstable. 19 packetdrill/gtests/net/tcp/zerocopy_fastopen-server_ipv6. is unstable. 20 packetdrill/gtests/net/tcp/zerocopy_maxfrags_ipv6. is unstable. 21 packetdrill/gtests/net/tcp/zerocopy_small_ipv6. is unstable. 22 packetdrill/gtests/net/tcp/zerocopy_batch_ipv6. is unstable. 23 packetdrill/gtests/net/tcp/zerocopy_epoll_edge_ipv6. is unstable. 24 packetdrill/gtests/net/tcp/zerocopy_fastopen-client_ipv6. is unstable. 25 packetdrill/gtests/net/tcp/zerocopy_epoll_exclusive_ipv6. is unstable. 26 packetdrill/gtests/net/tcp/blocking_blocking-connect_ipv4-mapped-v6. is unstable. 27 packetdrill/gtests/net/tcp/cwnd_moderation_cwnd-moderation-ecn-enter-cwr-no-moderation-700_ipv6. is unstable. 28 packetdrill/gtests/net/tcp/nagle_sockopt_cork_nodelay_ipv6. is unstable. 29 packetdrill/gtests/net/tcp/sendfile_sendfile-simple_ipv6. is unstable. 30 packetdrill/gtests/net/tcp/shutdown_shutdown-rdwr-write-queue-close_ipv4-mapped-v6. is unstable. 31 packetdrill/gtests/net/tcp/blocking_blocking-write_ipv4. is unstable. 32 packetdrill/gtests/net/tcp/shutdown_shutdown-rdwr-close_ipv6. is unstable. 33 packetdrill/gtests/net/tcp/sack_sack-shift-sacked-2-6-8-3-9-nofack_ipv6. is unstable. 34 packetdrill/gtests/net/tcp/shutdown_shutdown-rdwr-write-queue-close_ipv6. is unstable. 35 packetdrill/gtests/net/tcp/syscall_bad_arg_fastopen-invalid-buf-ptr_ipv6. is unstable. 36 packetdrill/gtests/net/tcp/syscall_bad_arg_syscall-invalid-buf-ptr_ipv6. is unstable. 37 packetdrill/gtests/net/tcp/tcp_info_tcp-info-sndbuf-limited_ipv4. is unstable. 38 packetdrill/gtests/net/tcp/timestamping_server_ipv6. is unstable. 39 packetdrill/gtests/net/tcp/ts_recent_reset_tsval_ipv6. is unstable. 40 packetdrill/gtests/net/tcp/shutdown_shutdown-rd-close_ipv6. is unstable. 41 packetdrill/gtests/net/tcp/shutdown_shutdown-wr-close_ipv6. is unstable. 42 packetdrill/gtests/net/tcp/splice_tcp_splice_loop_test_ipv6. sometimes may not run. 43 packetdrill/gtests/net/tcp/timestamping_server_ipv4. is unstable. 44 packetdrill/gtests/net/tcp/timestamping_partial_ipv6. is unstable. 45 packetdrill/gtests/net/tcp/validate_validate-established-no-flags_ipv6. is unstable. 46 packetdrill/gtests/net/tcp/user_timeout_user_timeout_ipv6. is unstable. 47 packetdrill/gtests/net/tcp/zerocopy_client_ipv6. is unstable. 48 packetdrill/gtests/net/tcp/user_timeout_user_timeout_ipv4-mapped-v6. is unstable. 49 packetdrill/gtests/net/tcp/blocking_blocking-read_ipv6. is unstable. 50 packetdrill/gtests/net/tcp/sack_sack-shift-sacked-7-3-4-8-9-fack_ipv6. is unstable. 51 packetdrill/gtests/net/tcp/sack_sack-shift-sacked-7-5-6-8-9-fack_ipv6. is unstable. 52 packetdrill/gtests/net/tcp/inq_server_ipv6. is unstable. 53 packetdrill/gtests/net/tcp/tcp_info_tcp-info-last_data_recv_ipv4-mapped-v6. is unstable. 54 packetdrill/gtests/net/tcp/blocking_blocking-read_ipv4-mapped-v6. is unstable. 55 packetdrill/gtests/net/tcp/md5_md5-only-on-client-ack_ipv6. is unstable. 56 packetdrill/gtests/net/tcp/tcp_info_tcp-info-rwnd-limited_ipv4. is unstable. 57 packetdrill/gtests/net/tcp/ts_recent_invalid_ack_ipv6. is unstable. 58 packetdrill/gtests/net/tcp/tcp_info_tcp-info-last_data_recv_ipv6. is unstable. 59 packetdrill/gtests/net/tcp/tcp_info_tcp-info-sndbuf-limited_ipv6. is unstable. 60 packetdrill/gtests/net/tcp/timestamping_partial_ipv4. is unstable. 61 packetdrill/gtests/net/tcp/mss_mss-getsockopt-tcp_maxseg-server_ipv6. is unstable. 62 packetdrill/gtests/net/tcp/syscall_bad_arg_sendmsg-empty-iov_ipv6. is unstable. 63 packetdrill/gtests/net/tcp/ts_recent_fin_tsval_ipv6. is unstable. 64 packetdrill/gtests/net/tcp/tcp_info_tcp-info-last_data_recv_ipv4. is unstable. 65 packetdrill/gtests/net/tcp/user_timeout_user_timeout_ipv4. is unstable. 66 packetdrill/gtests/net/tcp/blocking_blocking-read_ipv4. is unstable. 67 packetdrill/gtests/net/tcp/sack_sack-shift-sacked-7-3-4-8-9-fack_ipv4-mapped-v6. is unstable. 68 packetdrill/gtests/net/tcp/ts_recent_fin_tsval_ipv4-mapped-v6. is unstable. 69 packetdrill/gtests/net/tcp/nagle_sendmsg_msg_more_ipv4-mapped-v6. is unstable. 70 packetdrill/gtests/net/tcp/blocking_blocking-write_ipv6. is unstable. 71 packetdrill/gtests/net/tcp/close_close-remote-fin-then-close_ipv4-mapped-v6. is unstable. 72 packetdrill/gtests/net/tcp/blocking_blocking-accept_ipv6. is unstable. 73 packetdrill/gtests/net/tcp/gro_gro-mss-option_ipv6. is unstable. 74 packetdrill/gtests/net/tcp/user_timeout_user-timeout-probe_ipv4-mapped-v6. is unstable. 75 packetdrill/gtests/net/tcp/user_timeout_user-timeout-probe_ipv4. is unstable. 76 packetdrill/gtests/net/tcp/tcp_info_tcp-info-rwnd-limited_ipv6. is unstable.

[ligurio]

@CharileBrown, could you provide details about your test environment (operating system version, applied network settings etc) and packetdrill version?

[ligurio]

See also a note in https://github.com/google/packetdrill/commit/51a1a06b2cb38bce959691791a6b1a75ca7b3e27:

Note that some tests may be flaky due to timing constraints. One way of addressing that is with --tolerance_usecs. We may refine tests after initial commit.

[nealcardwell]

As noted above, more details (kernel version, test output for failed tests) would be helpful.

Could you please use the run_all.py script in the packetdrill distribution, and report the output? You can run it with something like:

cd ./packetdrill/gtests/net/ ./packetdrill/run_all.py -S -l -v tcp

This will run all tests in a serialized fashion, and log the error messages for failed tests.

For documentation on flags see: ./packetdrill/run_all.py --help

Thanks!