Retention taxonomy

[RichieB2B]

This retention taxonomy can be used to tag events and attributes that need to be expired after a certain amount of time. A companion MISP API scipt can clear the appropriate to_ids flags when the tagged IOC's become expired.

See also https://github.com/MISP/MISP/pull/4661

[adulau]

Thank you very much for the contribution. It's now merged.

I did the following changes:

• MANIFEST file has been updated to include the retention taxonomy
• Remove the hide_tag to make the schema validator happy

Open points

• Adding the hide_tag in the default taxonomy format (@adulau) (and change MISP to take this into account this (@iglocska)
• Review this taxonomy to be added in the MISP decaying models https://github.com/MISP/misp-decaying-models (@mokaddem)

{'value': 'expired', 'expanded': 'Set when the retention period has expired', 'numerical_value': 0, 'hide_tag': True}: Additional properties are not allowed ('hide_tag' was unexpected)

[mokaddem]

IMHO, The idea is good if the user is willing to do the tagging by himself, explicitly stating that no matter what, the tagged attribute should be marked as decayed once the elapsed time has been reached.

For an automated integration and support of this taxonomy in MISP, some questions may arise:

1. If the tagging is done automatically by MISP, what decaying model should be used to pick the correct tag?
2. What happens when an attribute has expired, should the tag be removed?
3. What happens if the IOC gets a sighting, should the tag be updated?
   • And more particularly, what if the attribute has already been expired, should the ids_flag be turned on again?

I see some huge challenges to support this model automatically.

However, we could only support this taxonomy during the score computation process: We could consider the tag in the computation and filter out expired attributes. That is the easiest way to accomplish this as we will have all the information at hand. In addition, having the possibility to quickly and easily set the expiration time on an IOC via the UI is really great for users!

My only fear with the taxonomy itself, is that users massively start adding automatically these kind of tags and fill up their database instead of using the built-in system (thus bypassing it when trying to do the decay themselves) because it will probably appear simpler.

tl;dr

We can support this taxonomy in the decaying branch by only using it to filter out tagged attributes.

[Linow974]

Hello, I did not quite understand how this script worked. I tested to activate the "retention" taxonomy, only I have "retention:1m" and "retention:expired" tags on one and only one event (out of 2000) and I don't know why. From how long does the "expired" tag appear?

[RichieB2B]

@Linow974 When you run the misp_retention.py script it will look for MISP events that are older than their "retention" tag. For example when an event with tag "retention:1m" is older than 1 month it will match. On all matched events the to_ids flag of IP based IOC's will be removed and a new tag "retention:expired" will be added. Both tags will remain on the event until they are manually removed.

I hope that clears things up.

[Linow974]

@Linow974 When you run the misp_retention.py script it will look for MISP events that are older than their "retention" tag. For example when an event with tag "retention:1m" is older than 1 month it will match. On all matched events the to_ids flag of IP based IOC's will be removed and a new tag "retention:expired" will be added. Both tags will remain on the event until they are manually removed.

I hope that clears things up.

Ok thanks, I understand better. And is this based on the last modification date or the creation date?

[RichieB2B]

It uses the "date" field of the event. This can be arbitrarily chosen by the creator of the event and usually indicates the day the event occurred.

https://github.com/MISP/MISP/blob/d2ddef68ab9abf01ae0e24e0d3d6bb196a80cfea/tools/misp_retention.py#L61

[Linow974]

Ok but the problem is that many of our events are updated, for example added attribute, added comment, etc. So an event may to be expired even though it was updated a few minutes ago.

[RichieB2B]

You can always change the script to use the modification date instead.