search

m-lab / ndt-server

docker native ndt5 and ndt7 server with prometheus integration
https://www.measurementlab.net/
Apache License 2.0
101 stars 40 forks source link

ndt5: zero length files created without content #307

Open stephen-soltesz opened 4 years ago

stephen-soltesz commented 4 years ago

@gfr10598 reported that gardener was observing about ~100k zero length files / day.

I spot checked a random ndt pod and found a number of zero-length files. Filtering only the zero length files:

/var/spool/ndt/ndt5/2020/07/01 # ls -al | grep ' 0 Jul'
-rw-r--r--    1 root     root             0 Jul  1 20:17 ndt-vbvrp_1593212104_000000000002CC65.json
-rw-r--r--    1 root     root             0 Jul  1 20:22 ndt-vbvrp_1593212104_000000000002CCFE.json
-rw-r--r--    1 root     root             0 Jul  1 20:24 ndt-vbvrp_1593212104_000000000002CD47.json
-rw-r--r--    1 root     root             0 Jul  1 20:34 ndt-vbvrp_1593212104_000000000002CE3E.json
-rw-r--r--    1 root     root             0 Jul  1 20:35 ndt-vbvrp_1593212104_000000000002CE7C.json
-rw-r--r--    1 root     root             0 Jul  1 20:36 ndt-vbvrp_1593212104_000000000002CE9A.json
-rw-r--r--    1 root     root             0 Jul  1 20:36 ndt-vbvrp_1593212104_000000000002CE9F.json
-rw-r--r--    1 root     root             0 Jul  1 20:41 ndt-vbvrp_1593212104_000000000002CF42.json
-rw-r--r--    1 root     root             0 Jul  1 20:43 ndt-vbvrp_1593212104_000000000002CF7C.json

The logic that writes the ndt5 result doesn't leave a lot of room between creating the file and writing the results, so it's not immediately clear how this could happen.

https://github.com/m-lab/ndt-server/blob/cbf50006359b991cfcfe3dfda2804dc2171fb506/ndt5/ndt5.go#L52-L63

stephen-soltesz commented 4 years ago

Well, the logs actually report the "Could not encode" error path. So, why can't it encode a simple struct? The actual value of err is not included in the log message.

2020/07/01 20:17:39 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 168.253.119.180 51446 2020-07-01 20:17:16.933302314 +0000 UTC m=+496.737844177 2020-07-01 20:17:39.202182801 +0000 UTC m=+519.006724664 0xc000314230 <nil> 0xc0001f20c0} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CC65.json
2020/07/01 20:22:08 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 129.56.91.165 49182 2020-07-01 20:21:52.736564444 +0000 UTC m=+772.541106307 2020-07-01 20:22:08.270110452 +0000 UTC m=+788.074652358 0xc000394140 <nil> 0xc000494180} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CCFE.json
2020/07/01 20:24:24 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 109.116.233.121 59913 2020-07-01 20:24:11.939008351 +0000 UTC m=+911.743550214 2020-07-01 20:24:24.005817325 +0000 UTC m=+923.810359219 0xc0003940f0 <nil> 0xc000746180} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CD47.json
2020/07/01 20:34:36 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 196.178.99.156 58166 2020-07-01 20:34:25.359860275 +0000 UTC m=+1525.164402138 2020-07-01 20:34:36.980160249 +0000 UTC m=+1536.784702146 0xc00012a1e0 <nil> 0xc00039a000} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CE3E.json
2020/07/01 20:35:51 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 196.44.110.52 28316 2020-07-01 20:35:40.251231651 +0000 UTC m=+1600.055773514 2020-07-01 20:35:51.438271434 +0000 UTC m=+1611.242813333 0xc00012a280 <nil> 0xc0001f2240} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CE7C.json
2020/07/01 20:36:50 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 158.148.226.122 15970 2020-07-01 20:36:38.299140074 +0000 UTC m=+1658.103681937 2020-07-01 20:36:50.228598651 +0000 UTC m=+1670.033140571 0xc0000b81e0 <nil> 0xc0001f2240} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CE9F.json
2020/07/01 20:36:59 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 154.160.6.14 20283 2020-07-01 20:36:32.68881886 +0000 UTC m=+1652.493360723 2020-07-01 20:36:59.000867308 +0000 UTC m=+1678.805409191 0xc0003940f0 <nil> 0xc000746180} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CE9A.json
2020/07/01 20:41:34 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 93.49.241.253 54357 2020-07-01 20:41:22.448514938 +0000 UTC m=+1942.253056801 2020-07-01 20:41:34.708802412 +0000 UTC m=+1954.513344335 0xc00012a320 <nil> 0xc00075e0c0} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CF42.json
2020/07/01 20:43:44 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 5.91.111.212 10162 2020-07-01 20:43:32.592839809 +0000 UTC m=+2072.397381672 2020-07-01 20:43:44.447492265 +0000 UTC m=+2084.252034162 0xc000394140 <nil> 0xc000746180} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002CF7C.json
2020/07/01 20:48:17 ndt5.go:61: Could not encode &{7e0ee25 v0.19.0 41.231.21.37 3010 105.112.131.122 55762 2020-07-01 20:48:01.718892818 +0000 UTC m=+2341.523434673 2020-07-01 20:48:17.996106672 +0000 UTC m=+2357.800648569 0xc00012a190 <nil> 0xc000746240} to /var/spool/ndt/ndt5/2020/07/01/ndt-vbvrp_1593212104_000000000002D003.json