search

purseclab / ATLAS

ATLAS: A Sequence-based Learning Approach for Attack Investigation
Apache License 2.0
137 stars 53 forks source link

What does "No skill and Lines" mean in evaluate.py? #3

Closed lesleychou closed 2 years ago

lesleychou commented 2 years ago

Hi Alsaheel,

Sorry for coming again. After I did an evaluation, I am a bit confused what are the following metrics:

No Skill: ROC AUC=0.500
Logistic: ROC AUC=0.850
Lines No Skill: ROC AUC=0.500
Lines Logistic: ROC AUC=0.951

I think the Logistic is the algorithm in NLP. Then what about the No skill and Lines? Could you give me an example of what they represent?

Thanks!

lesleychou commented 2 years ago

Oh, do the first two represent the results based on entity, and the last two based on event? If so, I am still a little confused about what "No skill" means?

cssaheel commented 2 years ago

Hi,

Yes, the first ones for entities, while Lines are meant for events. The no-skill line is a plotted straight line in standard ROC figures, it shows that if your classifier curve is below that no-skill line than it is a bad classifier.