search

purseclab / ATLAS

ATLAS: A Sequence-based Learning Approach for Attack Investigation
Apache License 2.0
137 stars 53 forks source link

The results of evaluate.py about entity #9

Closed zlsfe closed 2 years ago

zlsfe commented 2 years ago

After running evaluate.py, I got the result about entity as follow. It seems different from the result in your paper. Could you tell me what's wrong with it? Thanks！

Info (entity) Number of unique entities: 652 Number of malicious entities: 11 Result (entity) TP: 11 TN: 641 FP: 0 FN: 0

image

cssaheel commented 2 years ago

It seems evaluate.py for S1 does not show the right numbers, the evaluate.py shows smaller number than the actual numbers. The reason for this failure is that if you read the generated evaluation file: output/eval_seq_graph_testing_preprocessed_logs_S1-CVE-2015-5122_windows.dot.txt.json

You will see that some entities are lumped together, thus, many entities can be grouped as one entity like this "Entity1;Entity2; ...": c:/users/aalsahee/appdata/roaming/microsoft/windows/recent/customdestinations/atd2u3m1o6uzd4q4dugw.temp;c:/users/aalsahee/appdata/roaming/mozilla/firefox/profiles/a6gl280b.default/sessionstore.js.tmp;c:/users/aalsahee/a....

evaluate.py still counts these lumped entites as one entity! that's why these numbers generated by evaluate.py are smaller than the actual numbers. During writing the paper I must have implemented another script to consider the numbers of these lumped entities. I don't have it anymore, but evaluate.py can be easily fixed to consider the numbers of these lumped entities. I hope this helps. Thanks.