search

NCSC-NL / taranis3

Taranis
Other
59 stars 17 forks source link

abnormal behavior of items when clustered after linked to Analysis #17

Closed bsi-lz closed 4 years ago

bsi-lz commented 5 years ago

We observed that items, that are already linked to an Analysis (status "waitingroom") are switched to status "read" after the collector clustered them with fresh collected items. Other items in that cluster are set to "read" aswell. Is this normal behavior?

How do you handle such clusters? Our workarround is to uncluster and link each single item to analysis manually.

markov2 commented 5 years ago

After consulting the process specialist, I think I understand why the current workflow suits @NCSC, but my personal expectations are with you.

We monitor the incoming items close to continuously. Every 20 minutes, new items enter and get clustered. They are usually all analyzed before the next cycle starts. New items about the same subject which show-up in a later cycle are simply "too late" hence not studied anymore: marked as "read".

IMHO, the analysis would work like a dossier: the clustering should suggest to add late arrivals to the analysis which already contains items it the same cluster. This does has advantages, but code has not evolved that way. Also, it may simply be extra manual work without much benefit.

bsi-lz commented 5 years ago

We monitor the incoming items close to continuously. Every 20 minutes, new items enter and get clustered. They are usually all analyzed before the next cycle starts. New items about the same subject which show-up in a later cycle are simply "too late" hence not studied anymore: marked as "read".

Thanks for that insight.

IMHO, the analysis would work like a dossier: the clustering should suggest to add late arrivals to the analysis which already contains items it the same cluster.

That would be great :) But i guess you are not gonna change that behavior?

bsi-lz commented 5 years ago

If i wanted to change the behavior myself - clustering ignores "waitingroom" items - where do i begin? Can you give me a hint in the right direction? My guess would be Clustering.pm...

markov2 commented 5 years ago

I have submitted my view on how it should work as comment to SK's design spec for Taranis NG. I did not get response on that.

The change you need might be not too hard: exclude 'waitingroom' items from the selected items which are the input for clustering. Probably in the query of pm/Clustering.pm line 122. However, I have no chance to think and test such change through at the moment. Patches may get accepted ;-)

markov2 commented 4 years ago

no new info, closing