First step in expanding pcap parser

gfr10598 commented 2 years ago

This PR adds code to the pcap parser that uses gopacket to parse the top level packets from the pcap rawcontent. It does not parse the IP or TCP headers, as the gopacket implementation is rather heavyweight. Two new histogram metrics are introduced, in the pcap.go file (which is debatable) to track the connection duration and packet counts. These may not be important long term, but have been useful for development.

During development, I managed to cause some instance restarts with array index panics, so I added the panic handling to the active polling code. I also fixed the very very noisy labelling for the FileCount metric.

This change is

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 6860

71 of 93 (76.34%) changed or added relevant lines in 3 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.1%) to 63.835%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
active/poller.go	19	30	63.33%
parser/pcap.go	51	62	82.26%
<!--	Total:	71	93	76.34%	-->

Totals
Change from base Build 6733:	0.1%
Covered Lines:	3818
Relevant Lines:	5981

💛 - Coveralls

gfr10598 commented 2 years ago

Prototype dashboard with the new metrics is here: https://grafana.mlab-sandbox.measurementlab.net/goto/8Jd3j4pnk?orgId=1

gfr10598 commented 2 years ago

The benchmark is fairly complex, and I think the stats are the average across the three files that it parses. BenchmarkGetPackets-8 2156 493474 ns/op 1181692 B/op 1892 allocs/op

The associated profile looks like this:

Type: cpu
Time: Nov 22, 2021 at 11:26am (EST)
Duration: 2.22s, Total samples = 11.39s (513.48%)
Showing nodes accounting for 10.50s, 92.19% of 11.39s total
Dropped 137 nodes (cum <= 0.06s)
      flat  flat%   sum%        cum   cum%
     0.02s  0.18%  0.18%      5.27s 46.27%  github.com/m-lab/etl/parser.GetPackets
         0     0%  0.18%      5.27s 46.27%  github.com/m-lab/etl/parser_test.BenchmarkGetPackets.func1
         0     0%  0.18%      5.27s 46.27%  testing.(*B).RunParallel.func1
     0.01s 0.088%  0.26%      3.47s 30.47%  runtime.systemstack
         0     0%  0.26%      2.80s 24.58%  runtime.lock (inline)
     0.03s  0.26%  0.53%      2.80s 24.58%  runtime.lock2
         0     0%  0.53%      2.80s 24.58%  runtime.lockWithRank (inline)
     0.05s  0.44%  0.97%      2.77s 24.32%  github.com/google/gopacket/pcapgo.(*Reader).ReadPacketData
     0.01s 0.088%  1.05%      2.44s 21.42%  runtime.newstack
     0.01s 0.088%  1.14%      2.40s 21.07%  runtime.morestack
     0.01s 0.088%  1.23%      2.35s 20.63%  io.ReadFull (inline)
     0.03s  0.26%  1.49%      2.34s 20.54%  io.ReadAtLeast
     0.04s  0.35%  1.84%      2.31s 20.28%  compress/gzip.(*Reader).Read
         0     0%  1.84%      2.21s 19.40%  runtime.gopreempt_m
         0     0%  1.84%      2.21s 19.40%  runtime.goschedImpl
     0.08s   0.7%  2.55%      2.02s 17.73%  runtime.mallocgc
         0     0%  2.55%      1.90s 16.68%  runtime.growslice

profile006

gfr10598 commented 2 years ago

Changing the packet array allocation improves performance significantly

BenchmarkGetPackets-8       2156        493474  ns/op   1181692 B/op        1892 allocs/op
BenchmarkGetPackets-8       3165        335201  ns/op    510861 B/op        1887 allocs/op

Type: cpu
Time: Nov 22, 2021 at 11:38am (EST)
Duration: 1.21s, Total samples = 6.11s (504.80%)
Showing nodes accounting for 5.71s, 93.45% of 6.11s total
Dropped 87 nodes (cum <= 0.03s)
      flat  flat%   sum%        cum   cum%
         0     0%     0%      2.76s 45.17%  testing.(*B).RunParallel.func1
     0.02s  0.33%  0.33%      2.75s 45.01%  github.com/m-lab/etl/parser.GetPackets
         0     0%  0.33%      2.75s 45.01%  github.com/m-lab/etl/parser_test.BenchmarkGetPackets.func1
     0.03s  0.49%  0.82%      2.08s 34.04%  github.com/google/gopacket/pcapgo.(*Reader).ReadPacketData
     0.01s  0.16%  0.98%      1.96s 32.08%  io.ReadFull (inline)
     0.02s  0.33%  1.31%      1.95s 31.91%  io.ReadAtLeast
     0.05s  0.82%  2.13%      1.93s 31.59%  compress/gzip.(*Reader).Read
         0     0%  2.13%      1.68s 27.50%  runtime.lock (inline)
     0.02s  0.33%  2.45%      1.68s 27.50%  runtime.lock2
         0     0%  2.45%      1.68s 27.50%  runtime.lockWithRank (inline)
     0.02s  0.33%  2.78%      1.57s 25.70%  compress/flate.(*decompressor).Read
         0     0%  2.78%      1.52s 24.88%  runtime.morestack
         0     0%  2.78%      1.52s 24.88%  runtime.newstack
         0     0%  2.78%      1.47s 24.06%  runtime.systemstack
     0.30s  4.91%  7.69%      1.41s 23.08%  compress/flate.(*decompressor).huffmanBlock
         0     0%  7.69%      1.39s 22.75%  runtime.gopreempt_m
         0     0%  7.69%      1.39s 22.75%  runtime.goschedImpl
        1s 16.37% 24.06%         1s 16.37%  runtime.procyield
         0     0% 24.06%      0.74s 12.11%  runtime.gcBgMarkWorker.func2
     0.06s  0.98% 25.04%      0.74s 12.11%  runtime.gcDrain
         0     0% 25.04%      0.72s 11.78%  runtime.schedule
         0     0% 25.04%      0.69s 11.29%  runtime.mstart
         0     0% 25.04%      0.66s 10.80%  runtime.osyield (inline)
     0.65s 10.64% 35.68%      0.66s 10.80%  runtime.usleep
     0.40s  6.55% 42.23%      0.62s 10.15%  compress/flate.(*decompressor).huffSym
     0.01s  0.16% 42.39%      0.58s  9.49%  runtime.gcBgMarkWorker
         0     0% 42.39%      0.54s  8.84%  runtime.mcall
         0     0% 42.39%      0.53s  8.67%  runtime.park_m
         0     0% 42.39%      0.51s  8.35%  runtime.preemptM
     0.51s  8.35% 50.74%      0.51s  8.35%  runtime.pthread_kill
         0     0% 50.74%      0.51s  8.35%  runtime.signalM (inline)
     0.05s  0.82% 51.55%      0.50s  8.18%  runtime.mallocgc
     0.03s  0.49% 52.05%      0.46s  7.53%  runtime.makeslice

m-lab / etl

First step in expanding pcap parser #1031

Pull Request Test Coverage Report for Build 6860

💛 - Coveralls