FDUDSDE / MAGIC

Codes and data for USENIX Security 24 paper "MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning"
MIT License
64 stars 10 forks source link

Question about streamspot_parser #17

Closed wlynn00 closed 2 months ago

wlynn00 commented 2 months ago

Hi, I found the edge_type_dict defined in the file streamspot_parser.py as below: edge_type_dict = ['i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 't', 'u', 'v', 'w', 'y', 'z', 'A', 'C', 'D', 'E', 'G']

But there are more edge types in the streamspot dataset, can I know the motivation and reason for considering only some of them?

Jimmyokok commented 2 months ago

The thing is I have forgotten why pre-defined edge_type_dict is needed at all :). To examine what will happen if all edge types are considered, I have first determined all available edge types, which gives me this:

['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Then, I re-generate the streamspot dataset using the above edge types and re-produce the training and evaluation procedure on the streamspot dataset, which yields the following result:

AUC: 1.0
F1: 0.9999999995
PRECISION: 1.0
RECALL: 1.0
TN: 100
FN: 0
TP: 100
FP: 0
#Test_AUC: 1.0000±0.0000

and average performance under 10 random seeds:

AUC: 0.99985+0.0003201562118716488
F1: 0.9985123880522941+0.0031725634638853995
PRECISION: 0.9970491166763736+0.006288242946478126
RECALL: 1.0+0.0
TN: 99.7+0.6403124237432849
FN: 0.0+0.0
TP: 100.0+0.0
FP: 0.3+0.6403124237432849
#Test_AUC: 0.9999±0.0003

which indicates that the pre-defined edge_type_dict is useless at all and can be safely removed.

wlynn00 commented 2 months ago

I got it. Thanks!