Sandbox gfr new jitter - Githubissues

gfr10598 commented 2 years ago

This draft includes a lot of optimized packet decoding logic. Explicit overlay of the raw packet data, and explicit byte swapping of multi-byte fields improves performance by approximate 2X overall. Since a large part of processing is the simple decompression of the pcap data, this 2X improvement represents a much larger improvement in efficiency in the interpretation of the data.

This branch is roughly at parity with the previous work based on the standard gopacket decoders, with maps for sequence numbers, and processing of the TSVal and TSEcn values.

It still needs work on Sack interpretation, inflight computation, better detection of lost packets (lost to PCAP), and general validation.

This change is

gfr10598 commented 2 years ago

Compare with https://github.com/m-lab/etl/pull/1028#issuecomment-981279510

File: etl_worker
Build ID: 9fcfa1c5d86e7b2a549d201ab9f257d4ac9414f2
Type: cpu
Time: Nov 28, 2021 at 10:59pm (EST)
Duration: 30.18s, Total samples = 4.89mins (972.16%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top40 -cum
Showing nodes accounting for 148.19s, 50.51% of 293.37s total
Dropped 480 nodes (cum <= 1.47s)
Showing top 40 nodes out of 133
      flat  flat%   sum%        cum   cum%
         0     0%     0%    287.03s 97.84%  github.com/m-lab/etl/active.(*GardenerAPI).RunAll.func1
         0     0%     0%    287.03s 97.84%  github.com/m-lab/etl/active.(*throttledRunnable).Run
         0     0%     0%    287.03s 97.84%  github.com/m-lab/etl/worker.ProcessGKETask
         0     0%     0%    287.03s 97.84%  golang.org/x/sync/errgroup.(*Group).Go.func1
         0     0%     0%    287.03s 97.84%  main.(*runnable).Run
     0.02s 0.0068% 0.0068%    286.99s 97.83%  github.com/m-lab/etl/task.(*Task).ProcessAllTests
         0     0% 0.0068%    286.99s 97.83%  github.com/m-lab/etl/worker.DoGKETask
     0.01s 0.0034%  0.01%    189.69s 64.66%  github.com/m-lab/etl/parser.(*PCAPParser).ParseAndInsert
     1.80s  0.61%  0.62%    189.61s 64.63%  github.com/m-lab/etl/tcpip.ProcessPackets
     4.83s  1.65%  2.27%    118.56s 40.41%  github.com/m-lab/etl/tcpip.(*Summary).Add
     0.01s 0.0034%  2.27%     97.26s 33.15%  github.com/m-lab/etl/storage.(*GCSSource).NextTest
     0.01s 0.0034%  2.28%     97.04s 33.08%  bytes.(*Buffer).ReadFrom
         0     0%  2.28%     96.80s 33.00%  github.com/m-lab/etl/storage.(*GCSSource).nextData
         0     0%  2.28%     96.76s 32.98%  io/ioutil.ReadAll (inline)
         0     0%  2.28%     96.76s 32.98%  io/ioutil.readAll
     8.32s  2.84%  5.11%     94.92s 32.36%  github.com/m-lab/etl/tcp.(*State).Update
     0.03s  0.01%  5.12%     91.35s 31.14%  compress/gzip.(*Reader).Read
     0.05s 0.017%  5.14%     90.59s 30.88%  compress/flate.(*decompressor).Read
    16.91s  5.76% 10.90%     88.34s 30.11%  compress/flate.(*decompressor).huffmanBlock
    28.45s  9.70% 20.60%     79.02s 26.94%  runtime.mallocgc
    33.95s 11.57% 32.17%     57.89s 19.73%  compress/flate.(*decompressor).huffSym
     5.06s  1.72% 33.90%     42.66s 14.54%  runtime.makeslice
     4.42s  1.51% 35.41%     40.85s 13.92%  github.com/m-lab/etl/tcp.(*State).Option
     2.96s  1.01% 36.41%     40.67s 13.86%  github.com/m-lab/etl/tcpip.Wrap
    16.67s  5.68% 42.10%     38.07s 12.98%  bufio.(*Reader).ReadByte
     2.02s  0.69% 42.79%     34.14s 11.64%  runtime.newobject
     1.73s  0.59% 43.38%     30.06s 10.25%  github.com/m-lab/etl/tcp.(*Tracker).Sack
     0.04s 0.014% 43.39%     27.38s  9.33%  bufio.(*Reader).fill
     0.02s 0.0068% 43.40%     27.28s  9.30%  compress/flate.(*decompressor).nextBlock
         0     0% 43.40%     27.27s  9.30%  archive/tar.(*Reader).Read
         0     0% 43.40%     27.27s  9.30%  archive/tar.(*regFileReader).Read
     2.31s  0.79% 44.18%     25.79s  8.79%  github.com/google/gopacket/pcapgo.(*Reader).ReadPacketData
     0.29s 0.099% 44.28%     24.37s  8.31%  github.com/m-lab/go/logx.(*logEvery).Println
     1.42s  0.48% 44.77%     24.09s  8.21%  github.com/m-lab/go/logx.(*logEvery).ok
     0.70s  0.24% 45.00%     21.04s  7.17%  github.com/m-lab/etl/tcp.WrapTCP
     2.29s  0.78% 45.79%     17.41s  5.93%  github.com/m-lab/etl/tcp.(*TCPHeaderWrapper).parseTCPOptions
     0.07s 0.024% 45.81%     15.54s  5.30%  runtime.systemstack
    11.90s  4.06% 49.87%     15.10s  5.15%  runtime.heapBitsSetType
     1.29s  0.44% 50.31%     14.52s  4.95%  github.com/m-lab/etl/tcp.(*Tracker).checkSack
     0.61s  0.21% 50.51%     12.59s  4.29%  time.Now

newJitterNov28

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 6956

541 of 769 (70.35%) changed or added relevant lines in 5 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.6%) to 64.804%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
tcpip/tcpip.go	190	258	73.64%
tcp/tcp.go	344	504	68.25%
<!--	Total:	541	769	70.35%	-->

Totals
Change from base Build 6945:	0.6%
Covered Lines:	4338
Relevant Lines:	6694

💛 - Coveralls

gfr10598 commented 2 years ago

Currently seeing about 440 Mb/sec per instance in sandbox!! This is about 41% of the index only throughput.

stephen-soltesz commented 2 years ago

Closing old PR - and referencing the notes from this change in https://github.com/m-lab/etl/issues/1085

gfr10598 commented 2 years ago

Heh - nice cleanup. Sorry I've lost touch. Would enjoy talking sometime if you like.

On Thu, Jun 2, 2022 at 3:16 PM Stephen Soltesz @.***> wrote:

Closing old PR - and referencing the notes from this change in #1085 https://github.com/m-lab/etl/issues/1085

— Reply to this email directly, view it on GitHub https://github.com/m-lab/etl/pull/1033#issuecomment-1145223431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE7SM6S6ZYY5XSMRZIBOW33VNEB7ZANCNFSM5I6F6LPA . You are receiving this because you authored the thread.Message ID: @.***>

stephen-soltesz commented 2 years ago

@gfr10598 then don't be a stranger!

m-lab / etl

Sandbox gfr new jitter #1033

Pull Request Test Coverage Report for Build 6956

💛 - Coveralls