search

mozilla / neqo

Neqo, the Mozilla Firefox implementation of QUIC in Rust
https://firefox-source-docs.mozilla.org/networking/http/http3.html
Apache License 2.0
1.85k stars 124 forks source link

ci: Treat a QNS `null` result as a failure #2043

Closed larseggert closed 3 months ago

larseggert commented 3 months ago

null happens when a client or server is deemed "not compliant" by QNS

github-actions[bot] commented 3 months ago

Failed Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

github-actions[bot] commented 3 months ago

Benchmark results

Performance differences relative to 3d0efa27c3c0c5629af2b6fdc43524624c9697d9.

coalesce_acked_from_zero 1+1 entries: :broken_heart: Performance has regressed.
       time:   [194.40 ns 194.87 ns 195.37 ns]
       change: [+1.2335% +1.6940% +2.0993%] (p = 0.00 < 0.05)

Found 9 outliers among 100 measurements (9.00%)
  4 (4.00%) high mild
  5 (5.00%) high severe
coalesce_acked_from_zero 3+1 entries: :broken_heart: Performance has regressed.
       time:   [236.74 ns 237.33 ns 237.96 ns]
       change: [+1.5956% +1.9648% +2.3132%] (p = 0.00 < 0.05)

Found 16 outliers among 100 measurements (16.00%)
  1 (1.00%) low mild
  9 (9.00%) high mild
  6 (6.00%) high severe
coalesce_acked_from_zero 10+1 entries: :broken_heart: Performance has regressed.
       time:   [237.52 ns 238.47 ns 239.54 ns]
       change: [+2.0606% +2.4359% +2.8378%] (p = 0.00 < 0.05)

Found 12 outliers among 100 measurements (12.00%)
  3 (3.00%) low mild
  9 (9.00%) high severe
coalesce_acked_from_zero 1000+1 entries: Change within noise threshold.
       time:   [214.82 ns 215.07 ns 215.37 ns]
       change: [+0.2670% +0.9830% +1.6798%] (p = 0.00 < 0.05)

Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe
RxStreamOrderer::inbound_frame(): Change within noise threshold.
       time:   [119.91 ms 120.10 ms 120.40 ms]
       change: [-0.9501% -0.7850% -0.5507%] (p = 0.00 < 0.05)

Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe
transfer/pacing-false/varying-seeds: No change in performance detected.
       time:   [40.745 ms 42.779 ms 44.809 ms]
       change: [-1.6650% +5.6269% +12.791%] (p = 0.11 > 0.05)

Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild
transfer/pacing-true/varying-seeds: No change in performance detected.
       time:   [56.693 ms 59.333 ms 62.031 ms]
       change: [-3.6566% +3.4641% +11.158%] (p = 0.35 > 0.05)

Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild
transfer/pacing-false/same-seed: No change in performance detected.
       time:   [49.307 ms 50.620 ms 51.939 ms]
       change: [-2.5498% +1.6213% +5.7517%] (p = 0.44 > 0.05)
transfer/pacing-true/same-seed: No change in performance detected.
       time:   [66.671 ms 72.234 ms 77.783 ms]
       change: [-10.712% -0.4478% +10.493%] (p = 0.94 > 0.05)

Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) low mild
1-conn/1-100mb-resp (aka. Download)/client: No change in performance detected.
       time:   [167.31 ms 173.98 ms 181.92 ms]
       thrpt:  [549.69 MiB/s 574.79 MiB/s 597.70 MiB/s]
change:
       time:   [-3.2499% +1.5853% +7.0313%] (p = 0.56 > 0.05)
       thrpt:  [-6.5694% -1.5606% +3.3590%]

Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) high mild
  6 (6.00%) high severe
1-conn/10_000-parallel-1b-resp (aka. RPS)/client: No change in performance detected.
       time:   [408.36 ms 412.07 ms 415.80 ms]
       thrpt:  [24.050 Kelem/s 24.268 Kelem/s 24.488 Kelem/s]
change:
       time:   [-1.5029% -0.3013% +1.0175%] (p = 0.62 > 0.05)
       thrpt:  [-1.0072% +0.3022% +1.5259%]

Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild
1-conn/1-1b-resp (aka. HPS)/client: No change in performance detected.
       time:   [46.031 ms 46.743 ms 47.456 ms]
       thrpt:  [21.072  elem/s 21.394  elem/s 21.725  elem/s]
change:
       time:   [-0.7639% +1.5862% +3.8817%] (p = 0.16 > 0.05)
       thrpt:  [-3.7366% -1.5615% +0.7698%]

Client/server transfer results

Transfer of 33554432 bytes over loopback. Client Server CC Pacing Mean [ms] Min [ms] Max [ms] Relative
msquic msquic 135.2 ± 67.0 96.2 400.2 1.00
neqo msquic reno on 305.4 ± 60.6 258.1 456.6 1.00
neqo msquic reno 288.1 ± 46.9 257.6 424.2 1.00
neqo msquic cubic on 264.7 ± 8.2 252.4 278.6 1.00
neqo msquic cubic 294.3 ± 77.2 259.7 512.9 1.00
msquic neqo reno on 206.5 ± 76.4 149.5 402.2 1.00
msquic neqo reno 171.8 ± 57.4 126.3 337.3 1.00
msquic neqo cubic on 176.9 ± 69.7 137.4 397.8 1.00
msquic neqo cubic 210.2 ± 82.9 110.4 386.0 1.00
neqo neqo reno on 207.8 ± 88.0 161.5 500.3 1.00
neqo neqo reno 225.1 ± 75.2 148.2 391.6 1.00
neqo neqo cubic on 206.8 ± 59.7 152.5 360.5 1.00
neqo neqo cubic 191.9 ± 63.1 154.9 419.0 1.00

:arrow_down: Download logs

github-actions[bot] commented 3 months ago

Firefox builds for this PR

The following builds are available for testing. Crossed-out builds did not succeed.