search

network-quality / goresponsiveness

A draft-ietf-ippm-responsiveness client in Go.
GNU General Public License v2.0
132 stars 10 forks source link

Panic Unexpectedly reusing a connection! #45

Open nitzan-tz opened 1 year ago

nitzan-tz commented 1 year ago

I build the container from the latest main branch on Ubuntu 20.04 but each time it runs it crashes with panic on "Unexpectedly reusing a connection!"

This is my kernel Linux automation-ng 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@automation-ng:~/goresponsiveness# docker run --rm goresp --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config --debug
Configuration: Version: 1
Small URL: https://mensura.cdn-apple.com/api/v1/gm/small
Large URL: https://mensura.cdn-apple.com/api/v1/gm/large
Upload URL: https://mensura.cdn-apple.com/api/v1/gm/slurp
Endpoint: defra3-edge-bx-021.aaplimg.com

Test will end no later than 2023-03-21 15:21:53.871068901 +0000 UTC m=+10.001201548
03-21-2023 15:21:44 UTC Go Responsiveness to mensura.cdn-apple.com:443...
Started a load-generating upload (id: 1).
Timeout expected to end at 2023-03-21 15:21:53.871068901 +0000 UTC m=+10.001201548
upload: Sleeping until 2023-03-21 15:21:45.119829016 +0000 UTC m=+1.249961598
Started a load-generating download (id: 2).
download: Sleeping until 2023-03-21 15:21:45.11995679 +0000 UTC m=+1.250089335
Started getting connection for 2 @ 2023-03-21 15:21:44.120186611 +0000 UTC m=+0.250319141
DNS Start for 2: {defra3-edge-bx-021.aaplimg.com}
DNS Done for 2: {[{17.253.57.41 } {2a01:b740:a10:1000::b2 }] <nil> true}
TCP Start for 2 at 2023-03-21 15:21:44.173691488 +0000 UTC m=+0.303824013
(combined probe) About to send round 1 of probes!
(SelfUp Probe) Started getting connection for Probe 3 @ 2023-03-21 15:21:44.221114812 +0000 UTC m=+0.351247342
(Foreign Probe) Started getting connection for Probe 4 @ 2023-03-21 15:21:44.221129631 +0000 UTC m=+0.351262193
(SelfDown Probe) Started getting connection for Probe 5 @ 2023-03-21 15:21:44.221148274 +0000 UTC m=+0.351280834
(SelfUp Probe) DNS Start for Probe 3: {defra3-edge-bx-021.aaplimg.com}
(SelfDown Probe) DNS Start for Probe 5: {defra3-edge-bx-021.aaplimg.com}
(Foreign Probe) DNS Start for Probe 4: {defra3-edge-bx-021.aaplimg.com}
TCP Done for 2 (with error <nil>) @ 2023-03-21 15:21:44.225727701 +0000 UTC m=+0.355860277
Started TLS Handshake for 2 @ Some: true
(Foreign Probe) DNS Done for Probe 4: {[{17.253.57.41 } {2a01:b740:a10:1000::b2 }] <nil> true}
(SelfUp Probe) DNS Done for Probe 3: {[{17.253.57.41 } {2a01:b740:a10:1000::b2 }] <nil> true}
(SelfDown Probe) DNS Done for Probe 5: {[{17.253.57.41 } {2a01:b740:a10:1000::b2 }] <nil> true}
(Foreign Probe) TCP Start for Probe 4 at 2023-03-21 15:21:44.278458025 +0000 UTC m=+0.408590619
(SelfUp Probe) TCP Start for Probe 3 at 2023-03-21 15:21:44.278479194 +0000 UTC m=+0.408611719
(SelfDown Probe) TCP Start for Probe 5 at 2023-03-21 15:21:44.278525028 +0000 UTC m=+0.408657559
Completed TLS handshake for 2 at Some: true with info {772 true false 4865 h2 true mensura.cdn-apple.com [0xc0000e4b00 0xc0000e5080] [] [] [48 130 3 117 10 1 0 160 130 3 110 48 130 3 106 6 9 43 6 1 5 5 7 48 1 1 4 130 3 91 48 130 3 87 48 129 158 162 22 4 20 80 118 21 49 26 93 94 208 20 65 205 251 188 135 89 126 75 252 73 145 24 15 50 48 50 51 48 51 50 49 48 55 52 51 49 51 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 233 201 151 242 96 167 83 69 221 79 157 219 73 80 107 223 195 137 118 190 4 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 2 16 101 201 113 247 63 65 134 15 127 127 20 38 176 97 152 40 128 0 24 15 50 48 50 51 48 51 50 49 48 55 52 51 49 51 90 160 17 24 15 50 48 50 51 48 51 50 49 50 51 52 51 49 50 90 48 10 6 8 42 134 72 206 61 4 3 2 3 71 0 48 68 2 32 86 69 96 85 211 226 87 255 179 38 205 21 99 251 108 233 232 130 231 87 159 200 169 129 205 232 181 98 21 8 190 82 2 32 67 55 30 219 12 121 92 148 22 96 67 4 224 67 170 50 102 165 44 56 166 156 200 125 138 52 200 151 21 114 145 162 160 130 2 93 48 130 2 89 48 130 2 85 48 130 1 251 160 3 2 1 2 2 16 107 138 145 89 125 57 172 65 191 223 201 122 3 153 44 174 48 10 6 8 42 134 72 206 61 4 3 2 48 100 49 43 48 41 6 3 85 4 3 19 34 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 19 48 17 6 3 85 4 8 19 10 67 97 108 105 102 111 114 110 105 97 49 11 48 9 6 3 85 4 6 19 2 85 83 48 30 23 13 50 51 48 51 50 48 49 55 52 52 51 55 90 23 13 50 51 48 53 48 49 49 55 52 52 51 54 90 48 107 49 71 48 69 6 3 85 4 3 19 62 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 32 79 67 83 80 32 82 101 115 112 111 110 100 101 114 32 80 71 50 32 50 48 50 51 48 51 50 48 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 11 48 9 6 3 85 4 6 19 2 85 83 48 89 48 19 6 7 42 134 72 206 61 2 1 6 8 42 134 72 206 61 3 1 7 3 66 0 4 12 156 232 58 61 220 228 87 118 141 6 74 211 211 145 127 61 163 33 170 27 197 53 89 81 153 254 163 156 59 105 15 159 61 98 0 212 83 76 235 14 213 96 141 212 155 171 24 188 241 128 171 126 212 53 38 72 183 117 255 54 55 251 218 163 129 135 48 129 132 48 12 6 3 85 29 19 1 1 255 4 2 48 0 48 31 6 3 85 29 35 4 24 48 22 128 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 48 15 6 9 43 6 1 5 5 7 48 1 5 4 2 5 0 48 19 6 3 85 29 37 4 12 48 10 6 8 43 6 1 5 5 7 3 9 48 29 6 3 85 29 14 4 22 4 20 80 118 21 49 26 93 94 208 20 65 205 251 188 135 89 126 75 252 73 145 48 14 6 3 85 29 15 1 1 255 4 4 3 2 7 128 48 10 6 8 42 134 72 206 61 4 3 2 3 72 0 48 69 2 32 71 240 83 199 85 237 228 234 244 6 19 162 254 132 35 101 202 228 100 220 119 145 233 216 237 183 74 85 143 94 85 5 2 33 0 195 103 224 133 158 35 198 225 41 105 215 9 254 253 76 192 166 205 194 155 74 91 63 254 66 193 73 79 206 224 239 139] [] 0x615c60}
Started getting connection for 2 @ 2023-03-21 15:21:44.281545173 +0000 UTC m=+0.411677704
Unexpectedly reusing a connection!
A self probe sent used a new connection!
panic: false

goroutine 22 [running]:
github.com/network-quality/goresponsiveness/lgc.(*LoadGeneratingConnectionDownload).SetGotConnTimeInfo(0x0?, {0xc000317630?, 0x461f39?, 0x9e54c0?}, {{0x7f3a38?, 0xc000348000?}, 0x40?, 0x2a?, 0xc000317680?})
        /goresponsiveness/lgc/lgc.go:147 +0x1c8
github.com/network-quality/goresponsiveness/traceable.GenerateHttpTimingTracer.func6({{0x7f3a38?, 0xc000348000?}, 0xd0?, 0x53?, 0xc0003176e0?})
        /goresponsiveness/traceable/traceable.go:59 +0x8b
golang.org/x/net/http2.traceGotConn(0xc00010a300?, 0xc0000b8600, 0x1)
        /go/pkg/mod/golang.org/x/net@v0.7.0/http2/transport.go:3121 +0x1ec
golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc0000fc120, 0xc000308000, {0x20?})
        /go/pkg/mod/golang.org/x/net@v0.7.0/http2/transport.go:560 +0x1ac
golang.org/x/net/http2.(*Transport).RoundTrip(0xc0000f03c0?, 0xc00031e000?)
        /go/pkg/mod/golang.org/x/net@v0.7.0/http2/transport.go:513 +0x1b
net/http.(*Transport).roundTrip(0xc0000f03c0, 0xc000308000)
        /usr/local/go/src/net/http/transport.go:592 +0x7ba
net/http.(*Transport).RoundTrip(0x41922b?, 0x7f0580?)
        /usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc000308000, {0x7f0580, 0xc0000f03c0}, {0x7551e0?, 0x1?, 0x0?})
        /usr/local/go/src/net/http/client.go:252 +0x5d8
net/http.(*Client).send(0xc000098de0, 0xc000308000, {0x18?, 0x762080?, 0x0?})
        /usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc000098de0, 0xc000308000)
        /usr/local/go/src/net/http/client.go:725 +0x8f5
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:593
github.com/network-quality/goresponsiveness/lgc.(*LoadGeneratingConnectionDownload).doDownload(0xc0000fa840, {0x7f2f98?, 0xc00001c700})
        /goresponsiveness/lgc/lgc.go:333 +0x292
created by github.com/network-quality/goresponsiveness/lgc.(*LoadGeneratingConnectionDownload).Start
        /goresponsiveness/lgc/lgc.go:299 +0x2ad
root@automation-ng:~/goresponsiveness#
hawkinsw commented 1 year ago

Thank you for the feedback! I will investigate as soon as possible!

Will

hawkinsw commented 1 year ago

I can reproduce! I will investigate as soon as possible!

Thanks for reporting!

Will

hawkinsw commented 1 year ago

cc @critzo

hawkinsw commented 1 year ago

I am terribly sorry for the delay, @nitzan-tz !! I believe that https://github.com/network-quality/goresponsiveness/commit/e2bdd8109e9d707d30d518f4da334ddc2c747eb0 fixes the issue. If you could confirm, that would be awesome. Again, I am sorry for the delay!

I am cc'ing @critzo because they also reported a similar issue!

Will

hawkinsw commented 1 year ago

Given e2bdd81 and a0e0b18 and the success that we had with those helping another person who saw a similar bug, I am going to close this issue. We can always reopen!

nitzan-tz commented 1 year ago

Unfortunately, I still see panic but different one I did a new git clone root@containerlabs:~/goresponsiveness# git log commit a0e0b1861d5b2d0d77e728042c915f5b7742e744 (HEAD -> main, origin/main, origin/HEAD) Author: Will Hawkins <whh8b@obs.cr> Date: Tue Apr 25 22:55:58 2023 -0400

Create a new docker image root@containerlabs:~/goresponsiveness# docker image ls | grep gore goresp latest c8bfefd4cbb9 3 hours ago 416MB root@containerlabs:~/goresponsiveness#

and when I run it I get the following panic

`################# Responsiveness is instantaneously unstable.
download: Transferred: 11631848 bytes in 1.000102136s.
download: Load-generating connection with id 21 is in an unrecognizable state.
panic: download: Load-generating connection with id 21 is in an unrecognizable state.

goroutine 30 [running]:
github.com/network-quality/goresponsiveness/rpm.LoadGenerator.func1()
        /goresponsiveness/rpm/rpm.go:309 +0xfb9
created by github.com/network-quality/goresponsiveness/rpm.LoadGenerator
        /goresponsiveness/rpm/rpm.go:238 +0x1e5`

This is the full debug of the session

`root@containerlabs:~/goresponsiveness# docker run --rm goresp  -config mensura.cdn-apple.com  -port 443 -path /api/v1/gm/config  -debug
Configuration: Version: 1
Small URL: https://mensura.cdn-apple.com/api/v1/gm/small
Large URL: https://mensura.cdn-apple.com/api/v1/gm/large
Upload URL: https://mensura.cdn-apple.com/api/v1/gm/slurp
Endpoint: defra3-edge-bx-009.aaplimg.com

Test will end no later than 2023-05-12 18:40:28.250237728 +0000 UTC m=+10.001197428
05-12-2023 18:40:18 UTC Go Responsiveness to mensura.cdn-apple.com:443...
Timeout expected to end at 2023-05-12 18:40:28.250237728 +0000 UTC m=+10.001197428
Started a load-generating upload (id: 1).
upload: Sleeping until 2023-05-12 18:40:19.499864076 +0000 UTC m=+1.250823751
Started a load-generating download (id: 2).
download: Sleeping until 2023-05-12 18:40:19.499931813 +0000 UTC m=+1.250891488
Started getting connection for 2 @ 2023-05-12 18:40:18.500249843 +0000 UTC m=+0.251209523
DNS Start for 2: {defra3-edge-bx-009.aaplimg.com}
DNS Done for 2: {[{17.253.57.17 } {2a01:b740:a10:1000::52 }] <nil> true}
TCP Start for 2 at 2023-05-12 18:40:18.64848645 +0000 UTC m=+0.399446145
TCP Done for 2 (with error <nil>) @ 2023-05-12 18:40:18.704145032 +0000 UTC m=+0.455104707
Started TLS Handshake for 2 @ Some: true
Completed TLS handshake for 2 at Some: true with info {772 true false 4865 h2 true mensura.cdn-apple.com [0xc00026a000 0xc00026a580] [] [] [48 130 3 119 10 1 0 160 130 3 112 48 130 3 108 6 9 43 6 1 5 5 7 48 1 1 4 130 3 93 48 130 3 89 48 129 158 162 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 233 201 151 242 96 167 83 69 221 79 157 219 73 80 107 223 195 137 118 190 4 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 2 16 101 201 113 247 63 65 134 15 127 127 20 38 176 97 152 40 128 0 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 160 17 24 15 50 48 50 51 48 53 49 51 48 55 53 52 52 57 90 48 10 6 8 42 134 72 206 61 4 3 2 3 73 0 48 70 2 33 0 186 54 128 187 248 185 127 116 224 99 21 184 44 117 81 87 51 111 229 80 72 249 65 85 117 207 193 58 216 68 188 67 2 33 0 191 35 24 55 145 251 235 97 123 227 250 37 10 27 56 158 23 199 230 159 189 211 237 221 164 118 3 100 40 18 6 13 160 130 2 93 48 130 2 89 48 130 2 85 48 130 1 251 160 3 2 1 2 2 16 40 197 112 235 41 222 54 70 132 39 185 121 98 92 183 185 48 10 6 8 42 134 72 206 61 4 3 2 48 100 49 43 48 41 6 3 85 4 3 19 34 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 19 48 17 6 3 85 4 8 19 10 67 97 108 105 102 111 114 110 105 97 49 11 48 9 6 3 85 4 6 19 2 85 83 48 30 23 13 50 51 48 52 49 50 49 56 48 54 53 56 90 23 13 50 51 48 53 50 52 49 56 48 54 53 55 90 48 107 49 71 48 69 6 3 85 4 3 19 62 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 32 79 67 83 80 32 82 101 115 112 111 110 100 101 114 32 82 71 49 32 50 48 50 51 48 52 49 50 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 11 48 9 6 3 85 4 6 19 2 85 83 48 89 48 19 6 7 42 134 72 206 61 2 1 6 8 42 134 72 206 61 3 1 7 3 66 0 4 130 172 27 161 126 35 130 137 46 245 78 87 151 240 218 26 122 229 126 242 202 90 71 78 235 168 61 252 149 236 122 207 122 136 235 211 205 56 16 206 120 102 16 65 199 170 33 8 106 244 163 97 80 241 205 132 252 211 229 32 147 243 25 250 163 129 135 48 129 132 48 12 6 3 85 29 19 1 1 255 4 2 48 0 48 31 6 3 85 29 35 4 24 48 22 128 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 48 15 6 9 43 6 1 5 5 7 48 1 5 4 2 5 0 48 19 6 3 85 29 37 4 12 48 10 6 8 43 6 1 5 5 7 3 9 48 29 6 3 85 29 14 4 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 48 14 6 3 85 29 15 1 1 255 4 4 3 2 7 128 48 10 6 8 42 134 72 206 61 4 3 2 3 72 0 48 69 2 32 73 255 47 240 149 211 39 213 82 139 3 99 137 164 195 45 204 233 14 8 240 115 234 109 70 105 109 72 186 204 212 211 2 33 0 237 52 59 228 171 42 126 5 50 205 132 145 123 126 225 126 80 59 203 162 216 116 74 136 158 145 134 58 51 181 156 166] [] 0x615c60}
Got connection for 2 at 2023-05-12 18:40:18.764638455 +0000 UTC m=+0.515598118 with info {0xc000240000 false false 0s}
(lgd) Http finished writing request for 2 at 2023-05-12 18:40:18.764746964 +0000 UTC m=+0.515706640 with info {<nil>}
Got the first byte of HTTP response headers for 2 at 2023-05-12 18:40:18.820864889 +0000 UTC m=+0.571824564
(combined probe) About to send round 1 of probes!
(SelfUp Probe) Started getting connection for Probe 3 @ 2023-05-12 18:40:18.921720046 +0000 UTC m=+0.672679730
(SelfUp Probe) Got a reused connection for Probe 3 at 2023-05-12 18:40:18.92173931 +0000 UTC m=+0.672698985 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 4 @ 2023-05-12 18:40:18.921797779 +0000 UTC m=+0.672757476
(SelfDown Probe) Started getting connection for Probe 5 @ 2023-05-12 18:40:18.921785545 +0000 UTC m=+0.672745230
(SelfDown Probe) Got a reused connection for Probe 5 at 2023-05-12 18:40:18.921824407 +0000 UTC m=+0.672784093 with info {0xc000240000 true false 0s}.
(SelfUp Probe) Http finished writing request for Probe 3 at 2023-05-12 18:40:18.921861734 +0000 UTC m=+0.672821409 with info {<nil>}
(Foreign Probe) DNS Start for Probe 4: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Http finished writing request for Probe 5 at 2023-05-12 18:40:18.922061241 +0000 UTC m=+0.673020917 with info {<nil>}
(SelfUp Probe) Http response is ready for Probe 3 at 2023-05-12 18:40:18.978425588 +0000 UTC m=+0.729385272
(Probe 3): Http TLS and Header Time: 56.686287ms
(Probe 3): Http Download Time: 83.22µs
(combined probe) (SelfUp Probe 3) sanity vs total: 56.871015ms vs 56.769507ms
Probe Stabilizer: MA: 0.056770 ns (previous 4 intervals).
################# Responsiveness is instantaneously unstable.
(Foreign Probe) DNS Done for Probe 4: {[{17.253.57.17 } {2a01:b740:a10:1000::52 }] <nil> false}
(Foreign Probe) TCP Start for Probe 4 at 2023-05-12 18:40:18.979488018 +0000 UTC m=+0.730447694
(combined probe) About to send round 2 of probes!
(Foreign Probe) Started getting connection for Probe 7 @ 2023-05-12 18:40:19.021768834 +0000 UTC m=+0.772728513
(SelfDown Probe) Started getting connection for Probe 8 @ 2023-05-12 18:40:19.021778084 +0000 UTC m=+0.772737781
(SelfDown Probe) Got a reused connection for Probe 8 at 2023-05-12 18:40:19.021808608 +0000 UTC m=+0.772768283 with info {0xc000240000 true false 0s}.
(SelfUp Probe) Started getting connection for Probe 6 @ 2023-05-12 18:40:19.021822663 +0000 UTC m=+0.772782382
(Foreign Probe) DNS Start for Probe 7: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Got a reused connection for Probe 6 at 2023-05-12 18:40:19.021839112 +0000 UTC m=+0.772798810 with info {0xc00009b880 true false 0s}.
(SelfDown Probe) Http finished writing request for Probe 8 at 2023-05-12 18:40:19.021914908 +0000 UTC m=+0.772874584 with info {<nil>}
(SelfUp Probe) Http finished writing request for Probe 6 at 2023-05-12 18:40:19.021920981 +0000 UTC m=+0.772880657 with info {<nil>}
(Foreign Probe) TCP Done for Probe 4 (with error <nil>) @ 2023-05-12 18:40:19.034425914 +0000 UTC m=+0.785385599
(Foreign Probe) Started TLS Handshake for Probe 4 @ Some: true
(SelfUp Probe) Http response is ready for Probe 6 at 2023-05-12 18:40:19.079480997 +0000 UTC m=+0.830440674
(Probe 6): Http TLS and Header Time: 57.641864ms
(Probe 6): Http Download Time: 85.331µs
(combined probe) (SelfUp Probe 6) sanity vs total: 57.809219ms vs 57.727195ms
Probe Stabilizer: MA: 0.057009 ns (previous 4 intervals).
################# Responsiveness is instantaneously unstable.
(Foreign Probe) DNS Done for Probe 7: {[{17.253.57.17 } {2a01:b740:a10:1000::52 }] <nil> false}
(Foreign Probe) TCP Start for Probe 7 at 2023-05-12 18:40:19.07986185 +0000 UTC m=+0.830821526
(Foreign Probe) Completed TLS handshake for Probe 4 at Some: true with info {772 true false 4865 h2 true mensura.cdn-apple.com [0xc0003c0000 0xc0003c0580] [] [] [48 130 3 119 10 1 0 160 130 3 112 48 130 3 108 6 9 43 6 1 5 5 7 48 1 1 4 130 3 93 48 130 3 89 48 129 158 162 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 233 201 151 242 96 167 83 69 221 79 157 219 73 80 107 223 195 137 118 190 4 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 2 16 101 201 113 247 63 65 134 15 127 127 20 38 176 97 152 40 128 0 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 160 17 24 15 50 48 50 51 48 53 49 51 48 55 53 52 52 57 90 48 10 6 8 42 134 72 206 61 4 3 2 3 73 0 48 70 2 33 0 186 54 128 187 248 185 127 116 224 99 21 184 44 117 81 87 51 111 229 80 72 249 65 85 117 207 193 58 216 68 188 67 2 33 0 191 35 24 55 145 251 235 97 123 227 250 37 10 27 56 158 23 199 230 159 189 211 237 221 164 118 3 100 40 18 6 13 160 130 2 93 48 130 2 89 48 130 2 85 48 130 1 251 160 3 2 1 2 2 16 40 197 112 235 41 222 54 70 132 39 185 121 98 92 183 185 48 10 6 8 42 134 72 206 61 4 3 2 48 100 49 43 48 41 6 3 85 4 3 19 34 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 19 48 17 6 3 85 4 8 19 10 67 97 108 105 102 111 114 110 105 97 49 11 48 9 6 3 85 4 6 19 2 85 83 48 30 23 13 50 51 48 52 49 50 49 56 48 54 53 56 90 23 13 50 51 48 53 50 52 49 56 48 54 53 55 90 48 107 49 71 48 69 6 3 85 4 3 19 62 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 32 79 67 83 80 32 82 101 115 112 111 110 100 101 114 32 82 71 49 32 50 48 50 51 48 52 49 50 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 11 48 9 6 3 85 4 6 19 2 85 83 48 89 48 19 6 7 42 134 72 206 61 2 1 6 8 42 134 72 206 61 3 1 7 3 66 0 4 130 172 27 161 126 35 130 137 46 245 78 87 151 240 218 26 122 229 126 242 202 90 71 78 235 168 61 252 149 236 122 207 122 136 235 211 205 56 16 206 120 102 16 65 199 170 33 8 106 244 163 97 80 241 205 132 252 211 229 32 147 243 25 250 163 129 135 48 129 132 48 12 6 3 85 29 19 1 1 255 4 2 48 0 48 31 6 3 85 29 35 4 24 48 22 128 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 48 15 6 9 43 6 1 5 5 7 48 1 5 4 2 5 0 48 19 6 3 85 29 37 4 12 48 10 6 8 43 6 1 5 5 7 3 9 48 29 6 3 85 29 14 4 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 48 14 6 3 85 29 15 1 1 255 4 4 3 2 7 128 48 10 6 8 42 134 72 206 61 4 3 2 3 72 0 48 69 2 32 73 255 47 240 149 211 39 213 82 139 3 99 137 164 195 45 204 233 14 8 240 115 234 109 70 105 109 72 186 204 212 211 2 33 0 237 52 59 228 171 42 126 5 50 205 132 145 123 126 225 126 80 59 203 162 216 116 74 136 158 145 134 58 51 181 156 166] [] 0x615c60}
(Foreign Probe) Http finished writing request for Probe 4 at 2023-05-12 18:40:19.093659828 +0000 UTC m=+0.844619503 with info {<nil>}
(combined probe) About to send round 3 of probes!
(SelfUp Probe) Started getting connection for Probe 9 @ 2023-05-12 18:40:19.121882887 +0000 UTC m=+0.872842584
(SelfUp Probe) Got a reused connection for Probe 9 at 2023-05-12 18:40:19.121905437 +0000 UTC m=+0.872865113 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 10 @ 2023-05-12 18:40:19.121893295 +0000 UTC m=+0.872853011
(SelfDown Probe) Started getting connection for Probe 11 @ 2023-05-12 18:40:19.121974043 +0000 UTC m=+0.872933737
(SelfDown Probe) Got a reused connection for Probe 11 at 2023-05-12 18:40:19.122036701 +0000 UTC m=+0.872996377 with info {0xc000240000 true false 0s}.
(Foreign Probe) DNS Start for Probe 10: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http finished writing request for Probe 9 at 2023-05-12 18:40:19.122008025 +0000 UTC m=+0.872967701 with info {<nil>}
(SelfDown Probe) Http finished writing request for Probe 11 at 2023-05-12 18:40:19.122114721 +0000 UTC m=+0.873074396 with info {<nil>}
(SelfDown Probe) Http response is ready for Probe 5 at 2023-05-12 18:40:19.127661204 +0000 UTC m=+0.878620883
(Probe 5): Http TLS and Header Time: 205.83679ms
(Probe 5): Http Download Time: 5.363726ms
(combined probe) (SelfDown Probe 5) sanity vs total: 211.315474ms vs 211.200516ms
Probe Stabilizer: MA: 0.074195 ns (previous 4 intervals).
################# Responsiveness is instantaneously unstable.
(Foreign Probe) TCP Done for Probe 7 (with error <nil>) @ 2023-05-12 18:40:19.136754463 +0000 UTC m=+0.887714141
(Foreign Probe) Started TLS Handshake for Probe 7 @ Some: true
(Foreign Probe) Http response is ready for Probe 4 at 2023-05-12 18:40:19.149686091 +0000 UTC m=+0.900645773
(Probe 4): Http TLS and Header Time: 115.260174ms
(Probe 4): Http Download Time: 48.271µs
(Probe 4): TCP Connection Time: 54.937905ms
(combined probe) (Foreign Probe 4) sanity vs total: 228.027006ms vs 170.24635ms
Probe Stabilizer: MA: 0.079486 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.022958 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.003974316422916667 s).
Probe Stabilizer: Values: 0.056769507, 0.057248351, 0.10856573933333334, 0.0953617165, 
################# Responsiveness is instantaneously unstable.
(Foreign Probe) DNS Done for Probe 10: {[{17.253.57.17 } {2a01:b740:a10:1000::52 }] <nil> false}
(Foreign Probe) TCP Start for Probe 10 at 2023-05-12 18:40:19.183362855 +0000 UTC m=+0.934322531
(SelfUp Probe) Http response is ready for Probe 9 at 2023-05-12 18:40:19.192173984 +0000 UTC m=+0.943133660
(Probe 9): Http TLS and Header Time: 70.268547ms
(Probe 9): Http Download Time: 58.064µs
(combined probe) (SelfUp Probe 9) sanity vs total: 70.405891ms vs 70.326611ms
Probe Stabilizer: MA: 0.080330 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.022156 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.004016488088541667 s).
Probe Stabilizer: Values: 0.060143240249999994, 0.057248351, 0.10856573933333334, 0.0953617165, 
################# Responsiveness is instantaneously unstable.
(Foreign Probe) Completed TLS handshake for Probe 7 at Some: true with info {772 true false 4865 h2 true mensura.cdn-apple.com [0xc0003c0b00 0xc0003c1080] [] [] [48 130 3 119 10 1 0 160 130 3 112 48 130 3 108 6 9 43 6 1 5 5 7 48 1 1 4 130 3 93 48 130 3 89 48 129 158 162 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 233 201 151 242 96 167 83 69 221 79 157 219 73 80 107 223 195 137 118 190 4 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 2 16 101 201 113 247 63 65 134 15 127 127 20 38 176 97 152 40 128 0 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 160 17 24 15 50 48 50 51 48 53 49 51 48 55 53 52 52 57 90 48 10 6 8 42 134 72 206 61 4 3 2 3 73 0 48 70 2 33 0 186 54 128 187 248 185 127 116 224 99 21 184 44 117 81 87 51 111 229 80 72 249 65 85 117 207 193 58 216 68 188 67 2 33 0 191 35 24 55 145 251 235 97 123 227 250 37 10 27 56 158 23 199 230 159 189 211 237 221 164 118 3 100 40 18 6 13 160 130 2 93 48 130 2 89 48 130 2 85 48 130 1 251 160 3 2 1 2 2 16 40 197 112 235 41 222 54 70 132 39 185 121 98 92 183 185 48 10 6 8 42 134 72 206 61 4 3 2 48 100 49 43 48 41 6 3 85 4 3 19 34 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 19 48 17 6 3 85 4 8 19 10 67 97 108 105 102 111 114 110 105 97 49 11 48 9 6 3 85 4 6 19 2 85 83 48 30 23 13 50 51 48 52 49 50 49 56 48 54 53 56 90 23 13 50 51 48 53 50 52 49 56 48 54 53 55 90 48 107 49 71 48 69 6 3 85 4 3 19 62 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 32 79 67 83 80 32 82 101 115 112 111 110 100 101 114 32 82 71 49 32 50 48 50 51 48 52 49 50 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 11 48 9 6 3 85 4 6 19 2 85 83 48 89 48 19 6 7 42 134 72 206 61 2 1 6 8 42 134 72 206 61 3 1 7 3 66 0 4 130 172 27 161 126 35 130 137 46 245 78 87 151 240 218 26 122 229 126 242 202 90 71 78 235 168 61 252 149 236 122 207 122 136 235 211 205 56 16 206 120 102 16 65 199 170 33 8 106 244 163 97 80 241 205 132 252 211 229 32 147 243 25 250 163 129 135 48 129 132 48 12 6 3 85 29 19 1 1 255 4 2 48 0 48 31 6 3 85 29 35 4 24 48 22 128 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 48 15 6 9 43 6 1 5 5 7 48 1 5 4 2 5 0 48 19 6 3 85 29 37 4 12 48 10 6 8 43 6 1 5 5 7 3 9 48 29 6 3 85 29 14 4 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 48 14 6 3 85 29 15 1 1 255 4 4 3 2 7 128 48 10 6 8 42 134 72 206 61 4 3 2 3 72 0 48 69 2 32 73 255 47 240 149 211 39 213 82 139 3 99 137 164 195 45 204 233 14 8 240 115 234 109 70 105 109 72 186 204 212 211 2 33 0 237 52 59 228 171 42 126 5 50 205 132 145 123 126 225 126 80 59 203 162 216 116 74 136 158 145 134 58 51 181 156 166] [] 0x615c60}
(Foreign Probe) Http finished writing request for Probe 7 at 2023-05-12 18:40:19.209311404 +0000 UTC m=+0.960271087 with info {<nil>}
(combined probe) About to send round 4 of probes!
(SelfUp Probe) Started getting connection for Probe 12 @ 2023-05-12 18:40:19.221979363 +0000 UTC m=+0.972939062
(SelfUp Probe) Got a reused connection for Probe 12 at 2023-05-12 18:40:19.222002683 +0000 UTC m=+0.972962359 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 13 @ 2023-05-12 18:40:19.222037861 +0000 UTC m=+0.972997548
(Foreign Probe) DNS Start for Probe 13: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http finished writing request for Probe 12 at 2023-05-12 18:40:19.222081005 +0000 UTC m=+0.973040680 with info {<nil>}
(SelfDown Probe) Started getting connection for Probe 14 @ 2023-05-12 18:40:19.222226983 +0000 UTC m=+0.973186700
(SelfDown Probe) Got a reused connection for Probe 14 at 2023-05-12 18:40:19.222250745 +0000 UTC m=+0.973210535 with info {0xc000240000 true false 0s}.
(SelfDown Probe) Http finished writing request for Probe 14 at 2023-05-12 18:40:19.222533823 +0000 UTC m=+0.973493525 with info {<nil>}
(Foreign Probe) TCP Done for Probe 10 (with error <nil>) @ 2023-05-12 18:40:19.252670111 +0000 UTC m=+1.003629786
(Foreign Probe) Started TLS Handshake for Probe 10 @ Some: true
(SelfDown Probe) Http response is ready for Probe 8 at 2023-05-12 18:40:19.263269603 +0000 UTC m=+1.014229310
(Probe 8): Http TLS and Header Time: 241.461027ms
(Probe 8): Http Download Time: 7.66387ms
(combined probe) (SelfDown Probe 8) sanity vs total: 249.182644ms vs 249.124897ms
Probe Stabilizer: MA: 0.093077 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.019746 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.004653849559375001 s).
Probe Stabilizer: Values: 0.060143240249999994, 0.10823726866666666, 0.10856573933333334, 0.0953617165, 
################# Responsiveness is instantaneously unstable.
(Foreign Probe) Http response is ready for Probe 7 at 2023-05-12 18:40:19.278376484 +0000 UTC m=+1.029336160
(Probe 7): Http TLS and Header Time: 141.622019ms
(Probe 7): Http Download Time: 57.478µs
(Probe 7): TCP Connection Time: 56.892615ms
(combined probe) (Foreign Probe 7) sanity vs total: 256.725238ms vs 198.572112ms
Probe Stabilizer: MA: 0.093917 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.020446 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.0046958309708333345 s).
Probe Stabilizer: Values: 0.060143240249999994, 0.10823726866666666, 0.11192425225000001, 0.0953617165, 
################# Responsiveness is instantaneously unstable.
(Foreign Probe) DNS Done for Probe 13: {[{17.253.57.17 } {2a01:b740:a10:1000::52 }] <nil> false}
(Foreign Probe) TCP Start for Probe 13 at 2023-05-12 18:40:19.28647059 +0000 UTC m=+1.037430287
(SelfUp Probe) Http response is ready for Probe 12 at 2023-05-12 18:40:19.290469808 +0000 UTC m=+1.041429484
(Probe 12): Http TLS and Header Time: 68.467125ms
(Probe 12): Http Download Time: 63.19µs
(combined probe) (SelfUp Probe 12) sanity vs total: 68.590537ms vs 68.530315ms
Probe Stabilizer: MA: 0.086770 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.023465 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.004338504598958333 s).
Probe Stabilizer: Values: 0.060143240249999994, 0.10823726866666666, 0.11192425225000001, 0.06677560675000001, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 5 of probes!
(SelfUp Probe) Started getting connection for Probe 15 @ 2023-05-12 18:40:19.322207382 +0000 UTC m=+1.073167073
(SelfUp Probe) Got a reused connection for Probe 15 at 2023-05-12 18:40:19.322221452 +0000 UTC m=+1.073181128 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 16 @ 2023-05-12 18:40:19.322238717 +0000 UTC m=+1.073198411
(SelfDown Probe) Started getting connection for Probe 17 @ 2023-05-12 18:40:19.322235453 +0000 UTC m=+1.073195134
(SelfDown Probe) Got a reused connection for Probe 17 at 2023-05-12 18:40:19.322262893 +0000 UTC m=+1.073222569 with info {0xc000240000 true false 0s}.
(Foreign Probe) DNS Start for Probe 16: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Http finished writing request for Probe 17 at 2023-05-12 18:40:19.322325761 +0000 UTC m=+1.073285436 with info {<nil>}
(SelfUp Probe) Http finished writing request for Probe 15 at 2023-05-12 18:40:19.322511245 +0000 UTC m=+1.073470932 with info {<nil>}
(Foreign Probe) Completed TLS handshake for Probe 10 at Some: true with info {772 true false 4865 h2 true mensura.cdn-apple.com [0xc0003c1600 0xc0003c1b80] [] [] [48 130 3 119 10 1 0 160 130 3 112 48 130 3 108 6 9 43 6 1 5 5 7 48 1 1 4 130 3 93 48 130 3 89 48 129 158 162 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 233 201 151 242 96 167 83 69 221 79 157 219 73 80 107 223 195 137 118 190 4 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 2 16 101 201 113 247 63 65 134 15 127 127 20 38 176 97 152 40 128 0 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 160 17 24 15 50 48 50 51 48 53 49 51 48 55 53 52 52 57 90 48 10 6 8 42 134 72 206 61 4 3 2 3 73 0 48 70 2 33 0 186 54 128 187 248 185 127 116 224 99 21 184 44 117 81 87 51 111 229 80 72 249 65 85 117 207 193 58 216 68 188 67 2 33 0 191 35 24 55 145 251 235 97 123 227 250 37 10 27 56 158 23 199 230 159 189 211 237 221 164 118 3 100 40 18 6 13 160 130 2 93 48 130 2 89 48 130 2 85 48 130 1 251 160 3 2 1 2 2 16 40 197 112 235 41 222 54 70 132 39 185 121 98 92 183 185 48 10 6 8 42 134 72 206 61 4 3 2 48 100 49 43 48 41 6 3 85 4 3 19 34 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 19 48 17 6 3 85 4 8 19 10 67 97 108 105 102 111 114 110 105 97 49 11 48 9 6 3 85 4 6 19 2 85 83 48 30 23 13 50 51 48 52 49 50 49 56 48 54 53 56 90 23 13 50 51 48 53 50 52 49 56 48 54 53 55 90 48 107 49 71 48 69 6 3 85 4 3 19 62 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 32 79 67 83 80 32 82 101 115 112 111 110 100 101 114 32 82 71 49 32 50 48 50 51 48 52 49 50 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 11 48 9 6 3 85 4 6 19 2 85 83 48 89 48 19 6 7 42 134 72 206 61 2 1 6 8 42 134 72 206 61 3 1 7 3 66 0 4 130 172 27 161 126 35 130 137 46 245 78 87 151 240 218 26 122 229 126 242 202 90 71 78 235 168 61 252 149 236 122 207 122 136 235 211 205 56 16 206 120 102 16 65 199 170 33 8 106 244 163 97 80 241 205 132 252 211 229 32 147 243 25 250 163 129 135 48 129 132 48 12 6 3 85 29 19 1 1 255 4 2 48 0 48 31 6 3 85 29 35 4 24 48 22 128 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 48 15 6 9 43 6 1 5 5 7 48 1 5 4 2 5 0 48 19 6 3 85 29 37 4 12 48 10 6 8 43 6 1 5 5 7 3 9 48 29 6 3 85 29 14 4 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 48 14 6 3 85 29 15 1 1 255 4 4 3 2 7 128 48 10 6 8 42 134 72 206 61 4 3 2 3 72 0 48 69 2 32 73 255 47 240 149 211 39 213 82 139 3 99 137 164 195 45 204 233 14 8 240 115 234 109 70 105 109 72 186 204 212 211 2 33 0 237 52 59 228 171 42 126 5 50 205 132 145 123 126 225 126 80 59 203 162 216 116 74 136 158 145 134 58 51 181 156 166] [] 0x615c60}
(Foreign Probe) Http finished writing request for Probe 10 at 2023-05-12 18:40:19.324347509 +0000 UTC m=+1.075307185 with info {<nil>}
(combined probe) About to send round 6 of probes!
(SelfUp Probe) Started getting connection for Probe 18 @ 2023-05-12 18:40:19.422558949 +0000 UTC m=+1.173518648
(SelfUp Probe) Got a reused connection for Probe 18 at 2023-05-12 18:40:19.422577936 +0000 UTC m=+1.173537611 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 19 @ 2023-05-12 18:40:19.4225819 +0000 UTC m=+1.173541577
(SelfDown Probe) Started getting connection for Probe 20 @ 2023-05-12 18:40:19.422638898 +0000 UTC m=+1.173598581
(Foreign Probe) DNS Start for Probe 19: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http finished writing request for Probe 18 at 2023-05-12 18:40:19.422707162 +0000 UTC m=+1.173666837 with info {<nil>}
(SelfDown Probe) Got a reused connection for Probe 20 at 2023-05-12 18:40:19.422655896 +0000 UTC m=+1.173615572 with info {0xc000240000 true false 0s}.
(SelfDown Probe) Http finished writing request for Probe 20 at 2023-05-12 18:40:19.423001332 +0000 UTC m=+1.173961008 with info {<nil>}
download: Transferred: 3145512 bytes in 1.000268406s.
Started a load-generating download (id: 21).
download: Sleeping until 2023-05-12 18:40:20.500442267 +0000 UTC m=+2.251401946
Download Throughput Stabilizer: MA: 23.991912 Mbps (previous 4 intervals).
################# Download is instantaneously unstable.
Started getting connection for 21 @ 2023-05-12 18:40:19.500578127 +0000 UTC m=+1.251537820
upload: Transferred: 0 bytes in 1.000562024s.
Started a load-generating upload (id: 22).
upload: Sleeping until 2023-05-12 18:40:20.500611183 +0000 UTC m=+2.251570846
DNS Start for 21: {defra3-edge-bx-009.aaplimg.com}
Upload Throughput Stabilizer: MA: 0.000000 Mbps (previous 4 intervals).
################# Upload is instantaneously unstable.
(combined probe) About to send round 7 of probes!
(SelfUp Probe) Started getting connection for Probe 23 @ 2023-05-12 18:40:19.523018106 +0000 UTC m=+1.273977803
(SelfUp Probe) Got a reused connection for Probe 23 at 2023-05-12 18:40:19.523038614 +0000 UTC m=+1.273998290 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 24 @ 2023-05-12 18:40:19.523115896 +0000 UTC m=+1.274075571
(SelfDown Probe) Started getting connection for Probe 25 @ 2023-05-12 18:40:19.523138762 +0000 UTC m=+1.274098440
(SelfDown Probe) Got a reused connection for Probe 25 at 2023-05-12 18:40:19.523289868 +0000 UTC m=+1.274249543 with info {0xc000240000 true false 0s}.
(SelfUp Probe) Http finished writing request for Probe 23 at 2023-05-12 18:40:19.523141857 +0000 UTC m=+1.274101533 with info {<nil>}
(Foreign Probe) DNS Start for Probe 24: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Http finished writing request for Probe 25 at 2023-05-12 18:40:19.523372827 +0000 UTC m=+1.274332503 with info {<nil>}
(Foreign Probe) TCP Start for Probe 13 at 2023-05-12 18:40:19.586742383 +0000 UTC m=+1.337702046
(Foreign Probe) TCP Done for Probe 13 (with error dial tcp [2a01:b740:a10:1000::52]:443: connect: cannot assign requested address) @ 2023-05-12 18:40:19.586912448 +0000 UTC m=+1.337872124
(SelfDown Probe) Http response is ready for Probe 11 at 2023-05-12 18:40:19.605443737 +0000 UTC m=+1.356403454
(Probe 11): Http TLS and Header Time: 483.407077ms
(Probe 11): Http Download Time: 177.426µs
(combined probe) (SelfDown Probe 11) sanity vs total: 483.750914ms vs 483.584503ms
Probe Stabilizer: MA: 0.114515 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.037181 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.005725764802083334 s).
Probe Stabilizer: Values: 0.1711240565, 0.10823726866666666, 0.11192425225000001, 0.06677560675000001, 
################# Responsiveness is instantaneously unstable.
(SelfDown Probe) Http response is ready for Probe 14 at 2023-05-12 18:40:19.607110994 +0000 UTC m=+1.358070673
(Probe 14): Http TLS and Header Time: 384.860138ms
(Probe 14): Http Download Time: 328.808µs
(combined probe) (SelfDown Probe 14) sanity vs total: 385.259827ms vs 385.188946ms
Probe Stabilizer: MA: 0.150174 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.068916 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.00750871915625 s).
Probe Stabilizer: Values: 0.1711240565, 0.250873617, 0.11192425225000001, 0.06677560675000001, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 8 of probes!
(SelfUp Probe) Started getting connection for Probe 26 @ 2023-05-12 18:40:19.62321879 +0000 UTC m=+1.374178470
(SelfUp Probe) Got a reused connection for Probe 26 at 2023-05-12 18:40:19.623230196 +0000 UTC m=+1.374189871 with info {0xc00009b880 true false 0s}.
(SelfDown Probe) Started getting connection for Probe 27 @ 2023-05-12 18:40:19.62323851 +0000 UTC m=+1.374198218
(SelfDown Probe) Got a reused connection for Probe 27 at 2023-05-12 18:40:19.623253463 +0000 UTC m=+1.374213138 with info {0xc000240000 true false 0s}.
(SelfUp Probe) Http finished writing request for Probe 26 at 2023-05-12 18:40:19.623338482 +0000 UTC m=+1.374298166 with info {<nil>}
(Foreign Probe) Started getting connection for Probe 28 @ 2023-05-12 18:40:19.623580456 +0000 UTC m=+1.374540162
(SelfDown Probe) Http finished writing request for Probe 27 at 2023-05-12 18:40:19.623627117 +0000 UTC m=+1.374586816 with info {<nil>}
(Foreign Probe) DNS Start for Probe 28: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http response is ready for Probe 15 at 2023-05-12 18:40:19.640606143 +0000 UTC m=+1.391565846
(SelfUp Probe) Http response is ready for Probe 18 at 2023-05-12 18:40:19.640670986 +0000 UTC m=+1.391630674
(SelfUp Probe) Http response is ready for Probe 23 at 2023-05-12 18:40:19.640696928 +0000 UTC m=+1.391656626
(Probe 23): Http TLS and Header Time: 117.658336ms
(Probe 18): Http TLS and Header Time: 218.093063ms
(Probe 15): Http TLS and Header Time: 318.384718ms
(Probe 15): Http Download Time: 60.642µs
(combined probe) (SelfUp Probe 15) sanity vs total: 318.517292ms vs 318.44536ms
Probe Stabilizer: MA: 0.200678 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.092399 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.010033882015625002 s).
Probe Stabilizer: Values: 0.1711240565, 0.250873617, 0.313937281, 0.06677560675000001, 
(Probe 18): Http Download Time: 54.436µs
(combined probe) (SelfUp Probe 18) sanity vs total: 218.220783ms vs 218.147499ms
(Probe 23): Http Download Time: 39.738µs
(combined probe) (SelfUp Probe 23) sanity vs total: 117.754551ms vs 117.698074ms
################# Responsiveness is instantaneously unstable.
Probe Stabilizer: MA: 0.271819 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.068330 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.013590956643750002 s).
Probe Stabilizer: Values: 0.1711240565, 0.250873617, 0.313937281, 0.351341577, 
################# Responsiveness is instantaneously unstable.
Probe Stabilizer: MA: 0.294006 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.040958 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.014700280559375 s).
Probe Stabilizer: Values: 0.25986996975, 0.250873617, 0.313937281, 0.351341577, 
################# Responsiveness is instantaneously unstable.
(SelfUp Probe) Http response is ready for Probe 26 at 2023-05-12 18:40:19.705823417 +0000 UTC m=+1.456783111
(Probe 26): Http TLS and Header Time: 82.59324ms
(Probe 26): Http Download Time: 107.831µs
(combined probe) (SelfUp Probe 26) sanity vs total: 82.760927ms vs 82.701071ms
Probe Stabilizer: MA: 0.277349 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.062823 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.013867460359375 s).
Probe Stabilizer: Values: 0.25986996975, 0.184248001, 0.313937281, 0.351341577, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 9 of probes!
(SelfUp Probe) Started getting connection for Probe 29 @ 2023-05-12 18:40:19.724315408 +0000 UTC m=+1.475275092
(SelfUp Probe) Got a reused connection for Probe 29 at 2023-05-12 18:40:19.72434786 +0000 UTC m=+1.475307539 with info {0xc00009b880 true false 0s}.
(SelfDown Probe) Started getting connection for Probe 31 @ 2023-05-12 18:40:19.724317153 +0000 UTC m=+1.475276847
(SelfDown Probe) Got a reused connection for Probe 31 at 2023-05-12 18:40:19.724408559 +0000 UTC m=+1.475368235 with info {0xc000240000 true false 0s}.
(Foreign Probe) Started getting connection for Probe 30 @ 2023-05-12 18:40:19.724432713 +0000 UTC m=+1.475392400
(SelfDown Probe) Http finished writing request for Probe 31 at 2023-05-12 18:40:19.724506993 +0000 UTC m=+1.475466671 with info {<nil>}
(SelfUp Probe) Http finished writing request for Probe 29 at 2023-05-12 18:40:19.724455756 +0000 UTC m=+1.475415438 with info {<nil>}
(Foreign Probe) DNS Start for Probe 30: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Http response is ready for Probe 17 at 2023-05-12 18:40:19.739709587 +0000 UTC m=+1.490669288
(Probe 17): Http TLS and Header Time: 417.446719ms
(Probe 17): Http Download Time: 139.625µs
(combined probe) (SelfDown Probe 17) sanity vs total: 417.665515ms vs 417.586344ms
Probe Stabilizer: MA: 0.251123 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.063961 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.012556159934375 s).
Probe Stabilizer: Values: 0.25986996975, 0.184248001, 0.209033247, 0.351341577, 
################# Responsiveness is instantaneously unstable.
(SelfDown Probe) Http response is ready for Probe 20 at 2023-05-12 18:40:19.761278249 +0000 UTC m=+1.512237924
(Probe 20): Http TLS and Header Time: 338.622352ms
(Probe 20): Http Download Time: 67.804µs
(combined probe) (SelfDown Probe 20) sanity vs total: 338.791667ms vs 338.690156ms
Probe Stabilizer: MA: 0.223080 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.028799 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.0111540016125 s).
Probe Stabilizer: Values: 0.25986996975, 0.184248001, 0.209033247, 0.23916891125, 
################# Responsiveness is instantaneously unstable.
(SelfDown Probe) Http response is ready for Probe 25 at 2023-05-12 18:40:19.761625602 +0000 UTC m=+1.512585288
(Probe 25): Http TLS and Header Time: 238.335745ms
(Probe 25): Http Download Time: 105.126µs
(combined probe) (SelfDown Probe 25) sanity vs total: 238.693978ms vs 238.440871ms
Probe Stabilizer: MA: 0.225451 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.031949 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.011272559621875 s).
Probe Stabilizer: Values: 0.2693546105, 0.184248001, 0.209033247, 0.23916891125, 
################# Responsiveness is instantaneously unstable.
(SelfUp Probe) Http response is ready for Probe 29 at 2023-05-12 18:40:19.78072959 +0000 UTC m=+1.531689266
(Probe 29): Http TLS and Header Time: 56.381727ms
(Probe 29): Http Download Time: 65.51µs
(combined probe) (SelfUp Probe 29) sanity vs total: 56.536307ms vs 56.447237ms
Probe Stabilizer: MA: 0.245087 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.023650 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.012254349009374998 s).
Probe Stabilizer: Values: 0.2693546105, 0.262791152, 0.209033247, 0.23916891125, 
################# Responsiveness is instantaneously unstable.
(Foreign Probe) Http response is ready for Probe 10 at 2023-05-12 18:40:19.798435047 +0000 UTC m=+1.549394722
(Probe 10): Http TLS and Header Time: 545.764936ms
(Probe 10): Http Download Time: 60.965µs
(Probe 10): TCP Connection Time: 69.307255ms
(combined probe) (Foreign Probe 10) sanity vs total: 676.653244ms vs 615.133156ms
Probe Stabilizer: MA: 0.234802 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.040227 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.011740122149999999 s).
Probe Stabilizer: Values: 0.2693546105, 0.262791152, 0.16789509825, 0.23916891125, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 10 of probes!
(SelfUp Probe) Started getting connection for Probe 32 @ 2023-05-12 18:40:19.824393716 +0000 UTC m=+1.575353397
(SelfUp Probe) Got a reused connection for Probe 32 at 2023-05-12 18:40:19.824415228 +0000 UTC m=+1.575374904 with info {0xc00009b880 true false 0s}.
(SelfDown Probe) Started getting connection for Probe 34 @ 2023-05-12 18:40:19.824429044 +0000 UTC m=+1.575388736
(SelfDown Probe) Got a reused connection for Probe 34 at 2023-05-12 18:40:19.824465388 +0000 UTC m=+1.575425064 with info {0xc000240000 true false 0s}.
(Foreign Probe) Started getting connection for Probe 33 @ 2023-05-12 18:40:19.824460006 +0000 UTC m=+1.575419690
(SelfUp Probe) Http finished writing request for Probe 32 at 2023-05-12 18:40:19.824508021 +0000 UTC m=+1.575467696 with info {<nil>}
(Foreign Probe) DNS Start for Probe 33: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Http finished writing request for Probe 34 at 2023-05-12 18:40:19.824649959 +0000 UTC m=+1.575609635 with info {<nil>}
(SelfUp Probe) Http response is ready for Probe 32 at 2023-05-12 18:40:19.890221493 +0000 UTC m=+1.641181204
(Probe 32): Http TLS and Header Time: 65.8063ms
(Probe 32): Http Download Time: 96.85µs
(combined probe) (SelfUp Probe 32) sanity vs total: 65.966451ms vs 65.90315ms
Probe Stabilizer: MA: 0.217575 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.048561 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.010878749215625 s).
Probe Stabilizer: Values: 0.2693546105, 0.262791152, 0.16789509825, 0.1702590765, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 11 of probes!
(SelfUp Probe) Started getting connection for Probe 35 @ 2023-05-12 18:40:19.924487533 +0000 UTC m=+1.675447221
(SelfUp Probe) Got a reused connection for Probe 35 at 2023-05-12 18:40:19.924498753 +0000 UTC m=+1.675458428 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 36 @ 2023-05-12 18:40:19.924562044 +0000 UTC m=+1.675521731
(SelfDown Probe) Started getting connection for Probe 37 @ 2023-05-12 18:40:19.924546641 +0000 UTC m=+1.675506321
(SelfDown Probe) Got a reused connection for Probe 37 at 2023-05-12 18:40:19.924602511 +0000 UTC m=+1.675562186 with info {0xc000240000 true false 0s}.
(Foreign Probe) DNS Start for Probe 36: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http finished writing request for Probe 35 at 2023-05-12 18:40:19.924589906 +0000 UTC m=+1.675549584 with info {<nil>}
(SelfDown Probe) Http finished writing request for Probe 37 at 2023-05-12 18:40:19.924768405 +0000 UTC m=+1.675728081 with info {<nil>}
(SelfUp Probe) Http response is ready for Probe 35 at 2023-05-12 18:40:19.993498936 +0000 UTC m=+1.744458652
(Probe 35): Http TLS and Header Time: 69.000224ms
(Probe 35): Http Download Time: 138.927µs
(combined probe) (SelfUp Probe 35) sanity vs total: 69.189938ms vs 69.139151ms
Probe Stabilizer: MA: 0.184307 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.047257 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.009215351183333335 s).
Probe Stabilizer: Values: 0.13628276791666666, 0.262791152, 0.16789509825, 0.1702590765, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 12 of probes!
(SelfUp Probe) Started getting connection for Probe 38 @ 2023-05-12 18:40:20.025134879 +0000 UTC m=+1.776094584
(SelfUp Probe) Got a reused connection for Probe 38 at 2023-05-12 18:40:20.025160185 +0000 UTC m=+1.776119868 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 39 @ 2023-05-12 18:40:20.025182114 +0000 UTC m=+1.776141803
(Foreign Probe) DNS Start for Probe 39: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Started getting connection for Probe 40 @ 2023-05-12 18:40:20.025320309 +0000 UTC m=+1.776279998
(SelfDown Probe) Got a reused connection for Probe 40 at 2023-05-12 18:40:20.025327363 +0000 UTC m=+1.776287045 with info {0xc000240000 true false 0s}.
(SelfUp Probe) Http finished writing request for Probe 38 at 2023-05-12 18:40:20.025348939 +0000 UTC m=+1.776308642 with info {<nil>}
(SelfDown Probe) Http finished writing request for Probe 40 at 2023-05-12 18:40:20.025441441 +0000 UTC m=+1.776401116 with info {<nil>}
(SelfUp Probe) Http response is ready for Probe 38 at 2023-05-12 18:40:20.087190288 +0000 UTC m=+1.838150006
(Probe 38): Http TLS and Header Time: 62.030138ms
(Probe 38): Http Download Time: 88.322µs
(combined probe) (SelfUp Probe 38) sanity vs total: 62.216613ms vs 62.11846ms
Probe Stabilizer: MA: 0.143747 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.028318 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.007187352865625 s).
Probe Stabilizer: Values: 0.13628276791666666, 0.10055128658333333, 0.16789509825, 0.1702590765, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 13 of probes!
(Foreign Probe) Started getting connection for Probe 42 @ 2023-05-12 18:40:20.125531665 +0000 UTC m=+1.876491363
(SelfUp Probe) Started getting connection for Probe 41 @ 2023-05-12 18:40:20.125556594 +0000 UTC m=+1.876516318
(SelfDown Probe) Started getting connection for Probe 43 @ 2023-05-12 18:40:20.125577557 +0000 UTC m=+1.876537261
(SelfUp Probe) Got a reused connection for Probe 41 at 2023-05-12 18:40:20.125580081 +0000 UTC m=+1.876539772 with info {0xc00009b880 true false 0s}.
(SelfDown Probe) Got a reused connection for Probe 43 at 2023-05-12 18:40:20.125590706 +0000 UTC m=+1.876550382 with info {0xc000240000 true false 0s}.
(Foreign Probe) DNS Start for Probe 42: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Http finished writing request for Probe 43 at 2023-05-12 18:40:20.125684957 +0000 UTC m=+1.876644633 with info {<nil>}
(SelfUp Probe) Http finished writing request for Probe 41 at 2023-05-12 18:40:20.125947547 +0000 UTC m=+1.876907251 with info {<nil>}
(SelfDown Probe) Http response is ready for Probe 27 at 2023-05-12 18:40:20.175227209 +0000 UTC m=+1.926186892
(Probe 27): Http TLS and Header Time: 551.973754ms
(Probe 27): Http Download Time: 106.316µs
(combined probe) (SelfDown Probe 27) sanity vs total: 552.13357ms vs 552.08007ms
Probe Stabilizer: MA: 0.148601 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.033272 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.007430041734375001 s).
Probe Stabilizer: Values: 0.13628276791666666, 0.10055128658333333, 0.18731020775, 0.1702590765, 
################# Responsiveness is instantaneously unstable.
(SelfUp Probe) Http response is ready for Probe 41 at 2023-05-12 18:40:20.190159333 +0000 UTC m=+1.941119023
(Probe 41): Http TLS and Header Time: 64.579251ms
(Probe 41): Http Download Time: 54.936µs
(combined probe) (SelfUp Probe 41) sanity vs total: 64.723799ms vs 64.634187ms
Probe Stabilizer: MA: 0.152784 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.036616 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.007639215365625 s).
Probe Stabilizer: Values: 0.13628276791666666, 0.10055128658333333, 0.18731020775, 0.186992967, 
################# Responsiveness is instantaneously unstable.
(SelfDown Probe) Http response is ready for Probe 31 at 2023-05-12 18:40:20.209745441 +0000 UTC m=+1.960705135
(Probe 31): Http TLS and Header Time: 485.3369ms
(Probe 31): Http Download Time: 102.858µs
(combined probe) (SelfDown Probe 31) sanity vs total: 485.562049ms vs 485.439758ms
Probe Stabilizer: MA: 0.191481 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.067497 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.009574032251041668 s).
Probe Stabilizer: Values: 0.29106811875, 0.10055128658333333, 0.18731020775, 0.186992967, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 14 of probes!
(SelfUp Probe) Started getting connection for Probe 44 @ 2023-05-12 18:40:20.225597735 +0000 UTC m=+1.976557432
(SelfUp Probe) Got a reused connection for Probe 44 at 2023-05-12 18:40:20.225612792 +0000 UTC m=+1.976572468 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 45 @ 2023-05-12 18:40:20.225684081 +0000 UTC m=+1.976643779
(SelfDown Probe) Started getting connection for Probe 46 @ 2023-05-12 18:40:20.225692665 +0000 UTC m=+1.976652351
(SelfDown Probe) Got a reused connection for Probe 46 at 2023-05-12 18:40:20.22570529 +0000 UTC m=+1.976664965 with info {0xc000240000 true false 0s}.
(Foreign Probe) DNS Start for Probe 45: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http finished writing request for Probe 44 at 2023-05-12 18:40:20.225756501 +0000 UTC m=+1.976716211 with info {<nil>}
(SelfDown Probe) Http finished writing request for Probe 46 at 2023-05-12 18:40:20.225754117 +0000 UTC m=+1.976713792 with info {<nil>}
(SelfUp Probe) Http response is ready for Probe 44 at 2023-05-12 18:40:20.282612396 +0000 UTC m=+2.033572084
(Probe 44): Http TLS and Header Time: 56.999616ms
(Probe 44): Http Download Time: 101.199µs
(combined probe) (SelfUp Probe 44) sanity vs total: 57.17382ms vs 57.100815ms
Probe Stabilizer: MA: 0.238796 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.051647 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.0119398125125 s).
Probe Stabilizer: Values: 0.29106811875, 0.2898137075, 0.18731020775, 0.186992967, 
################# Responsiveness is instantaneously unstable.
(SelfDown Probe) Http response is ready for Probe 34 at 2023-05-12 18:40:20.28310612 +0000 UTC m=+2.034065804
(Probe 34): Http TLS and Header Time: 458.64074ms
(Probe 34): Http Download Time: 61.733µs
(combined probe) (SelfDown Probe 34) sanity vs total: 458.777453ms vs 458.702473ms
Probe Stabilizer: MA: 0.258586 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.042479 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.012929301268749999 s).
Probe Stabilizer: Values: 0.29106811875, 0.2898137075, 0.26646930825000004, 0.186992967, 
################# Responsiveness is instantaneously unstable.
(combined probe) About to send round 15 of probes!
(Foreign Probe) Started getting connection for Probe 48 @ 2023-05-12 18:40:20.32586011 +0000 UTC m=+2.076819791
(Foreign Probe) DNS Start for Probe 48: {defra3-edge-bx-009.aaplimg.com}
(SelfDown Probe) Started getting connection for Probe 49 @ 2023-05-12 18:40:20.325890114 +0000 UTC m=+2.076849793
(SelfDown Probe) Got a reused connection for Probe 49 at 2023-05-12 18:40:20.326110945 +0000 UTC m=+2.077070630 with info {0xc000240000 true false 0s}.
(SelfDown Probe) Http finished writing request for Probe 49 at 2023-05-12 18:40:20.326245764 +0000 UTC m=+2.077205440 with info {<nil>}
(SelfUp Probe) Started getting connection for Probe 47 @ 2023-05-12 18:40:20.32648862 +0000 UTC m=+2.077448309
(SelfUp Probe) Got a reused connection for Probe 47 at 2023-05-12 18:40:20.326511894 +0000 UTC m=+2.077471577 with info {0xc00009b880 true false 0s}.
(SelfUp Probe) Http finished writing request for Probe 47 at 2023-05-12 18:40:20.326690171 +0000 UTC m=+2.077649874 with info {<nil>}
(Foreign Probe) TCP Done for Probe 13 (with error <nil>) @ 2023-05-12 18:40:20.364343238 +0000 UTC m=+2.115302914
(Foreign Probe) Started TLS Handshake for Probe 13 @ Some: true
(SelfUp Probe) Http response is ready for Probe 47 at 2023-05-12 18:40:20.382970152 +0000 UTC m=+2.133929847
(Probe 47): Http TLS and Header Time: 56.45827ms
(Probe 47): Http Download Time: 148.67µs
(combined probe) (SelfUp Probe 47) sanity vs total: 56.681846ms vs 56.60694ms
Probe Stabilizer: MA: 0.277953 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? true; Standard Deviation: 0.012516 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.0138976703875 s).
Probe Stabilizer: Values: 0.29106811875, 0.2898137075, 0.26646930825000004, 0.26446249650000003, 
################# Responsiveness is instantaneously stable.
(SelfDown Probe) Http response is ready for Probe 37 at 2023-05-12 18:40:20.390511959 +0000 UTC m=+2.141471635
(Probe 37): Http TLS and Header Time: 465.909449ms
(Probe 37): Http Download Time: 63.591µs
(combined probe) (SelfDown Probe 37) sanity vs total: 466.092517ms vs 465.97304ms
Probe Stabilizer: MA: 0.270085 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? true; Standard Deviation: 0.011661 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.013504266615625003 s).
Probe Stabilizer: Values: 0.259595817, 0.2898137075, 0.26646930825000004, 0.26446249650000003, 
################# Responsiveness is instantaneously stable.
(Foreign Probe) Completed TLS handshake for Probe 13 at Some: true with info {772 true false 4865 h2 true mensura.cdn-apple.com [0xc000502580 0xc000502b00] [] [] [48 130 3 119 10 1 0 160 130 3 112 48 130 3 108 6 9 43 6 1 5 5 7 48 1 1 4 130 3 93 48 130 3 89 48 129 158 162 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 48 115 48 113 48 73 48 9 6 5 43 14 3 2 26 5 0 4 20 233 201 151 242 96 167 83 69 221 79 157 219 73 80 107 223 195 137 118 190 4 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 2 16 101 201 113 247 63 65 134 15 127 127 20 38 176 97 152 40 128 0 24 15 50 48 50 51 48 53 49 50 49 53 53 52 53 48 90 160 17 24 15 50 48 50 51 48 53 49 51 48 55 53 52 52 57 90 48 10 6 8 42 134 72 206 61 4 3 2 3 73 0 48 70 2 33 0 186 54 128 187 248 185 127 116 224 99 21 184 44 117 81 87 51 111 229 80 72 249 65 85 117 207 193 58 216 68 188 67 2 33 0 191 35 24 55 145 251 235 97 123 227 250 37 10 27 56 158 23 199 230 159 189 211 237 221 164 118 3 100 40 18 6 13 160 130 2 93 48 130 2 89 48 130 2 85 48 130 1 251 160 3 2 1 2 2 16 40 197 112 235 41 222 54 70 132 39 185 121 98 92 183 185 48 10 6 8 42 134 72 206 61 4 3 2 48 100 49 43 48 41 6 3 85 4 3 19 34 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 19 48 17 6 3 85 4 8 19 10 67 97 108 105 102 111 114 110 105 97 49 11 48 9 6 3 85 4 6 19 2 85 83 48 30 23 13 50 51 48 52 49 50 49 56 48 54 53 56 90 23 13 50 51 48 53 50 52 49 56 48 54 53 55 90 48 107 49 71 48 69 6 3 85 4 3 19 62 65 112 112 108 101 32 80 117 98 108 105 99 32 83 101 114 118 101 114 32 69 67 67 32 67 65 32 49 50 32 45 32 71 49 32 79 67 83 80 32 82 101 115 112 111 110 100 101 114 32 82 71 49 32 50 48 50 51 48 52 49 50 49 19 48 17 6 3 85 4 10 19 10 65 112 112 108 101 32 73 110 99 46 49 11 48 9 6 3 85 4 6 19 2 85 83 48 89 48 19 6 7 42 134 72 206 61 2 1 6 8 42 134 72 206 61 3 1 7 3 66 0 4 130 172 27 161 126 35 130 137 46 245 78 87 151 240 218 26 122 229 126 242 202 90 71 78 235 168 61 252 149 236 122 207 122 136 235 211 205 56 16 206 120 102 16 65 199 170 33 8 106 244 163 97 80 241 205 132 252 211 229 32 147 243 25 250 163 129 135 48 129 132 48 12 6 3 85 29 19 1 1 255 4 2 48 0 48 31 6 3 85 29 35 4 24 48 22 128 20 95 227 46 138 148 151 222 211 92 225 183 212 188 152 142 49 41 201 144 58 48 15 6 9 43 6 1 5 5 7 48 1 5 4 2 5 0 48 19 6 3 85 29 37 4 12 48 10 6 8 43 6 1 5 5 7 3 9 48 29 6 3 85 29 14 4 22 4 20 185 157 1 232 86 183 197 18 16 184 209 31 86 100 234 129 121 61 244 12 48 14 6 3 85 29 15 1 1 255 4 4 3 2 7 128 48 10 6 8 42 134 72 206 61 4 3 2 3 72 0 48 69 2 32 73 255 47 240 149 211 39 213 82 139 3 99 137 164 195 45 204 233 14 8 240 115 234 109 70 105 109 72 186 204 212 211 2 33 0 237 52 59 228 171 42 126 5 50 205 132 145 123 126 225 126 80 59 203 162 216 116 74 136 158 145 134 58 51 181 156 166] [] 0x615c60}
(Foreign Probe) Http finished writing request for Probe 13 at 2023-05-12 18:40:20.425514901 +0000 UTC m=+2.176474576 with info {<nil>}
(combined probe) About to send round 16 of probes!
(SelfUp Probe) Started getting connection for Probe 50 @ 2023-05-12 18:40:20.426010836 +0000 UTC m=+2.176970516
(SelfUp Probe) Got a reused connection for Probe 50 at 2023-05-12 18:40:20.426083937 +0000 UTC m=+2.177043634 with info {0xc00009b880 true false 0s}.
(Foreign Probe) Started getting connection for Probe 51 @ 2023-05-12 18:40:20.426067705 +0000 UTC m=+2.177027396
(SelfDown Probe) Started getting connection for Probe 52 @ 2023-05-12 18:40:20.426177766 +0000 UTC m=+2.177137474
(SelfDown Probe) Got a reused connection for Probe 52 at 2023-05-12 18:40:20.42623035 +0000 UTC m=+2.177190050 with info {0xc000240000 true false 0s}.
(Foreign Probe) DNS Start for Probe 51: {defra3-edge-bx-009.aaplimg.com}
(SelfUp Probe) Http finished writing request for Probe 50 at 2023-05-12 18:40:20.426259162 +0000 UTC m=+2.177218838 with info {<nil>}
(SelfDown Probe) Http finished writing request for Probe 52 at 2023-05-12 18:40:20.426428039 +0000 UTC m=+2.177387736 with info {<nil>}
(SelfDown Probe) Http response is ready for Probe 40 at 2023-05-12 18:40:20.428268579 +0000 UTC m=+2.179228255
(Probe 40): Http TLS and Header Time: 402.94121ms
(Probe 40): Http Download Time: 83.132µs
(combined probe) (SelfDown Probe 40) sanity vs total: 403.052706ms vs 403.024342ms
Probe Stabilizer: MA: 0.284151 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.035840 s; Is Normally Distributed? true; Standard Deviation Cutoff: 0.014207554006250001 s).
Probe Stabilizer: Values: 0.259595817, 0.34607669874999997, 0.26646930825000004, 0.26446249650000003, 
################# Responsiveness is instantaneously unstable.
(SelfDown Probe) Http response is ready for Probe 43 at 2023-05-12 18:40:20.441523085 +0000 UTC m=+2.192482760
(Probe 43): Http TLS and Header Time: 315.932378ms
(Probe 43): Http Download Time: 105.893µs
(combined probe) (SelfDown Probe 43) sanity vs total: 316.081431ms vs 316.038271ms
Probe Stabilizer: MA: 0.295136 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.035469 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.01475682075625 s).
Probe Stabilizer: Values: 0.259595817, 0.34607669874999997, 0.31041064825, 0.26446249650000003, 
################# Responsiveness is instantaneously unstable.
(SelfUp Probe) Http response is ready for Probe 50 at 2023-05-12 18:40:20.485321673 +0000 UTC m=+2.236281360
(Probe 50): Http TLS and Header Time: 59.237726ms
(Probe 50): Http Download Time: 108.386µs
(combined probe) (SelfUp Probe 50) sanity vs total: 59.460936ms vs 59.346112ms
Probe Stabilizer: MA: 0.306795 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.030832 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.015339732565625 s).
Probe Stabilizer: Values: 0.259595817, 0.34607669874999997, 0.31041064825, 0.31109544125, 
################# Responsiveness is instantaneously unstable.
(Foreign Probe) Http response is ready for Probe 13 at 2023-05-12 18:40:20.486649759 +0000 UTC m=+2.237609443
(Probe 13): Http TLS and Header Time: 122.306529ms
(Probe 13): Http Download Time: 575.452µs
(Probe 13): TCP Connection Time: 777.600868ms
(combined probe) (Foreign Probe 13) sanity vs total: 1.265269472s vs 900.482849ms
Probe Stabilizer: MA: 0.301885 ns (previous 4 intervals).
Probe Stabilizer: Is Stable? false; Standard Deviation: 0.038553 s; Is Normally Distributed? false; Standard Deviation Cutoff: 0.01509425035625 s).
Probe Stabilizer: Values: 0.23995724024999998, 0.34607669874999997, 0.31041064825, 0.31109544125, 
################# Responsiveness is instantaneously unstable.
download: Transferred: 11631848 bytes in 1.000102136s.
download: Load-generating connection with id 21 is in an unrecognizable state.
panic: download: Load-generating connection with id 21 is in an unrecognizable state.

goroutine 30 [running]:
github.com/network-quality/goresponsiveness/rpm.LoadGenerator.func1()
        /goresponsiveness/rpm/rpm.go:309 +0xfb9
created by github.com/network-quality/goresponsiveness/rpm.LoadGenerator
        /goresponsiveness/rpm/rpm.go:238 +0x1e5`
hawkinsw commented 1 year ago

Thank you! We are tracking this problem in #48!

hawkinsw commented 1 year ago

@nitzan-tz I believe that we have finally fixed the problem -- can you check the updates in #48 and see what you think??

hawkinsw commented 8 months ago

I have fixed another bug (00cd532) related to this issue and I hope that will resolve these panics. cc @richb-hanover and @moeller0 because they saw panics recently for a related reason.

richb-hanover commented 8 months ago

Using 65b039e, I get these results.

I'm also not sure how to understand the "Final RPM" numbers (P90 and Single Sided) compared to all the other RPM numbers reported in the output. Thanks

√ goresponsiveness-1 % docker run --rm goresp --config mensura.cdn-apple.com --port 443 --path /api/v1/gm/config
01-20-2024 01:15:30 UTC Go Responsiveness to mensura.cdn-apple.com:443...
Warning: Throughput stability could not be reached. Making the test 15 seconds longer to calculate speculative RPM results.
Warning: Throughput stability could not be reached. Making the test 15 seconds longer to calculate speculative RPM results.
Results:
========
Download:
    Note: Test did not run to stability, these results are estimates.
    Throughput: 48.730 Mbps (6.091 MBps), using 16 parallel connections.
    RPM: 139 (P90)
    RPM: 320 (Single-Sided 5% Trimmed Mean)
========
Upload:
    Note: Test did not run to stability, these results are estimates.
    Throughput: 34.595 Mbps (4.324 MBps), using 16 parallel connections.
    RPM: 205 (P90)
    RPM: 841 (Single-Sided 5% Trimmed Mean)
========
Final RPM: 125 (P90)
Final RPM: 1597 (Single-Sided 5% Trimmed Mean)
√ goresponsiveness-1 %
hawkinsw commented 8 months ago

Thank you! I am glad that we are not seeing the panics anymore. The RPM scores are all derived the same way. The only difference is the source of the measurements used in the derivation. The first two only take into consideration the measurements made when the test is doing upload and download tests (serially). Then, all those measurements are put in to one group and then a final RPM calculation is done. Note, however, that what I just described will change (maybe even as soon as this weekend) because it takes in to account too many measurements (beyond those that the spec says combine to form the RPM).

Let me know if that helps!! Thanks again for testing! Will

richb-hanover commented 8 months ago

I'm not sure I understand the explanation above: it would help to explain the difference between the "source of the measurements", and it's not intuitively obvious how the two P90 RPMs (down and up) combine to give a Final RPM of 125.

Perhaps a section could be added to the top of the README.md that explains the output in its final form. I'm happy to help word-smith. Thanks

PS Perhaps this conversation (interpreting results) could be moved to a new issue...

moeller0 commented 8 months ago

I think for the trimmed mean it should be close to the average in the time domain, but since RPM is in the frequency domain this is not obvious... Say upload has 900ms delay, and download 100ms, for a time domain average of 500: Upload: 900ms -> 600 RPM Download: 100ms -> 67 RPM Final: 500ms -> 120 RPM

As I mentioned before reporting reciprocals has its own challenges... and the current numbers do not seem to match...

hawkinsw commented 8 months ago

Hello!

Sorry I wasn't more clear. The test proceeds in steps and, at each step, is operating in a particular "direction". Those steps may run in parallel but, by default, they run serially. The direction is specified with respect to the source of the bulk data being transmitted. The "download" step of the test occurs when the server is sending back bulk data over HTTP (GET). The "upload" step of the test occurs when the client is sending data to the server over HTTP (PUT).

As each of those directional tests is occurring, the tool is taking measurements to determine whether the network has stabilized at its worst. As soon as it does, it stops that test direction. Once both of the directions have stopped, those measurements taken by each direction in order to determine whether the connection was at its "worst" are combined to form a third set of measurements.

At that point, there are three sets of measurements:

  1. From the download direction
  2. From the upload direction
  3. A set of measurements built with the combination of (1) and (2) (there are no new measurements in this set).

The client (if properly invoked with the right parameters), will give you information about the RPM calculation for all three sets of those measurements.

Let me know if that helps!

Sorry again that my earlier message was not helpful! Will

richb-hanover commented 8 months ago

And I wasn't clear enough, either. I understand the test process (I edited the initial version of the RFC).

My questions revolve around clarifying the test reports: I want to split this new discussion from this "panic" report (which I believe we can close). I'll create a new issue with my questions later today. Thanks.