yampelo / beagle

Beagle is an incident response and digital forensics tool which transforms security logs and data into graphs.
MIT License
1.27k stars 145 forks source link

Improve graph generation speed #62

Closed yampelo closed 5 years ago

yampelo commented 5 years ago

This should fix #44 #24

codecov[bot] commented 5 years ago

Codecov Report

Merging #62 into master will increase coverage by 0.3%. The diff coverage is 98%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master      #62     +/-   ##
=========================================
+ Coverage   77.96%   78.26%   +0.3%     
=========================================
  Files          50       50             
  Lines        2637     2655     +18     
=========================================
+ Hits         2056     2078     +22     
+ Misses        581      577      -4
Impacted Files Coverage Δ
beagle/nodes/node.py 97.56% <100%> (+1.13%) :arrow_up:
beagle/common/__init__.py 96.87% <100%> (+3.12%) :arrow_up:
beagle/backends/networkx.py 96.77% <95%> (+4.99%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 354d442...00a13a5. Read the comment docs.

yampelo commented 5 years ago

This change improves graph generation time drastically.

The following comparison is against a HX triage of size 30mb, with 899941 extracted nodes from the agent event inspector.

Before: 40 minutes, 20 seconds

2019-10-28T00:13:41 | beagle.datasources.hx_triage.__init__:53 | INFO | Setting up HXTriage for data/hx/too_big/huge_triage.mans
2019-10-28T00:13:43 | beagle.transformers.fireeye_hx_transformer.__init__:17 | INFO | Created FireEyeHX Transformer.
2019-10-28T00:14:33 | beagle.transformers.base_transformer.run:111 | INFO | Finished processing of events, created 899941 nodes.
2019-10-28T00:14:33 | beagle.backends.networkx.__init__:52 | INFO | Initialized NetworkX Backend
2019-10-28T00:14:33 | beagle.backends.networkx.graph:69 | INFO | Beginning graph generation.
2019-10-28T00:53:55 | beagle.backends.networkx.graph:75 | INFO | Completed graph generation.
2019-10-28T00:53:55 | beagle.backends.networkx.graph:76 | INFO | Graph contains 21461 nodes and 233650 edges.

After 1 Minute, 32 seconds:

2019-10-28T00:58:41 | beagle.datasources.hx_triage.__init__:53 | INFO | Setting up HXTriage for data/hx/too_big/huge_triage.mans
2019-10-28T00:58:42 | beagle.transformers.fireeye_hx_transformer.__init__:17 | INFO | Created FireEyeHX Transformer.
2019-10-28T00:59:33 | beagle.transformers.base_transformer.run:111 | INFO | Finished processing of events, created 899941 nodes.
2019-10-28T00:59:33 | beagle.backends.networkx.__init__:53 | INFO | Initialized NetworkX Backend
2019-10-28T00:59:33 | beagle.backends.networkx.graph:70 | INFO | Beginning graph generation.
2019-10-28T00:59:59 | beagle.backends.networkx.graph:110 | INFO | Completed graph generation.
2019-10-28T00:59:59 | beagle.backends.networkx.graph:111 | INFO | Graph contains 21461 nodes and 233650 edges.