gfr10598
commented
2 years ago File: etl_worker
Build ID: 77564c5e62367b71c386fbcb33cda445495a2195
Type: cpu
Time: Nov 25, 2021 at 10:20am (EST)
Duration: 30.21s, Total samples = 5.39mins (1069.73%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top50 -cum
Showing nodes accounting for 221.13s, 68.42% of 323.20s total
Dropped 530 nodes (cum <= 1.62s)
Showing top 50 nodes out of 126
flat flat% sum% cum cum%
0 0% 0% 292.03s 90.36% github.com/m-lab/etl/active.(*GardenerAPI).RunAll.func1
0 0% 0% 292.03s 90.36% github.com/m-lab/etl/active.(*throttledRunnable).Run
0 0% 0% 292.03s 90.36% github.com/m-lab/etl/worker.ProcessGKETask
0 0% 0% 292.03s 90.36% golang.org/x/sync/errgroup.(*Group).Go.func1
0 0% 0% 292.03s 90.36% main.(*runnable).Run
0 0% 0% 292.01s 90.35% github.com/m-lab/etl/task.(*Task).ProcessAllTests
0 0% 0% 292.01s 90.35% github.com/m-lab/etl/worker.DoGKETask
0 0% 0% 191.24s 59.17% github.com/m-lab/etl/storage.(*GCSSource).NextTest
0.02s 0.0062% 0.0062% 191.12s 59.13% bytes.(*Buffer).ReadFrom
0 0% 0.0062% 190.54s 58.95% github.com/m-lab/etl/storage.(*GCSSource).nextData
0 0% 0.0062% 190.45s 58.93% io/ioutil.ReadAll (inline)
0 0% 0.0062% 190.45s 58.93% io/ioutil.readAll
0.06s 0.019% 0.025% 177.24s 54.84% compress/gzip.(*Reader).Read
0.10s 0.031% 0.056% 175.79s 54.39% compress/flate.(*decompressor).Read
31.30s 9.68% 9.74% 169.66s 52.49% compress/flate.(*decompressor).huffmanBlock
63.52s 19.65% 29.39% 113.84s 35.22% compress/flate.(*decompressor).huffSym
0.02s 0.0062% 29.40% 100.73s 31.17% github.com/m-lab/etl/parser.(*PCAPParser).ParseAndInsert
3.15s 0.97% 30.37% 100.60s 31.13% github.com/m-lab/etl/tcpip.GetPackets
34.07s 10.54% 40.92% 80.09s 24.78% bufio.(*Reader).ReadByte
0.05s 0.015% 40.93% 59.76s 18.49% bufio.(*Reader).fill
0.01s 0.0031% 40.93% 59.68s 18.47% archive/tar.(*Reader).Read
0.04s 0.012% 40.95% 59.67s 18.46% archive/tar.(*regFileReader).Read
0.05s 0.015% 40.96% 59.14s 18.30% runtime.systemstack
8.41s 2.60% 43.56% 58.46s 18.09% runtime.mallocgc
0.02s 0.0062% 43.57% 58.16s 18.00% compress/flate.(*decompressor).nextBlock
2.84s 0.88% 44.45% 40.14s 12.42% github.com/google/gopacket/pcapgo.(*Reader).ReadPacketData
1.05s 0.32% 44.77% 38.70s 11.97% runtime.makeslice
15.62s 4.83% 49.61% 30.90s 9.56% runtime.scanobject
4.16s 1.29% 50.89% 30.32s 9.38% github.com/m-lab/etl/tcpip.Wrap
0 0% 50.89% 25.47s 7.88% runtime.gcBgMarkWorker
0 0% 50.89% 25.11s 7.77% runtime.gcBgMarkWorker.func2
0.55s 0.17% 51.06% 25.11s 7.77% runtime.gcDrain
20.11s 6.22% 57.29% 20.11s 6.22% runtime.memmove
0.58s 0.18% 57.47% 19.53s 6.04% runtime.newobject
0.61s 0.19% 57.65% 17.42s 5.39% github.com/m-lab/etl/tcpip.WrapTCP
2.10s 0.65% 58.30% 16.28s 5.04% io.ReadFull (partial-inline)
2.95s 0.91% 59.22% 14.18s 4.39% io.ReadAtLeast
0 0% 59.22% 13.87s 4.29% bytes.(*Buffer).grow
3.35s 1.04% 60.25% 13.77s 4.26% compress/flate.(*decompressor).moreBits
0.08s 0.025% 60.28% 13.28s 4.11% runtime.(*mcache).nextFree
0.01s 0.0031% 60.28% 12.96s 4.01% runtime.(*mcache).refill
6.68s 2.07% 62.35% 12.87s 3.98% compress/flate.(*dictDecoder).tryWriteCopy
10.12s 3.13% 65.48% 12.87s 3.98% runtime.findObject
0.69s 0.21% 65.69% 12.38s 3.83% runtime.(*mcentral).cacheSpan
5.83s 1.80% 67.50% 12.10s 3.74% github.com/google/gopacket/pcapgo.(*Reader).readPacketHeader
0.46s 0.14% 67.64% 11.40s 3.53% runtime.(*mspan).sweep
1.87s 0.58% 68.22% 11.36s 3.51% runtime.bulkBarrierPreWrite
0 0% 68.22% 11.04s 3.42% runtime.(*mheap).alloc
0.65s 0.2% 68.42% 10.93s 3.38% runtime.typedmemmove
0 0% 68.42% 10.75s 3.33% bytes.makeSlice
coveralls
stephen-soltesz
This PR adds a small library that decodes ip and tcp layer by creating overlays on the underlying byte data, using the unsafe library.
It uses the gopacket library for supporting types and constants.
This level of decoding incurs significant memory allocation, but performance is superior to gopacket library, so far. When hooked into the pcap parser, the throughput per 15 cpu instance drops from roughly 1.07 GB/sec to roughly 660 MB/sec.
This change is