Method to Purge Soft-Deleted Attributes

[github-germ]

Feature Request: Provide a server side method to expeditiously delete soft deleted attributes.

Note: This Feature Request is spun out of https://github.com/MISP/MISP/issues/6929#issuecomment-770023542 based on discussion with @iglocska

Current Implementation: Soft deleted attributes continue to grow in MISP via actions performed on the WebUI, PyMISP, and Feed Delta Merge Processing. Soft-deleted attributes can be deleted one by one via PyMISP; however, this is a costly method, and in a system where a huge amount of soft-deletes have accumulated (e.g. millions via Delta Merge), this process can take days to complete.

Possible Solutions:

• Perhaps moving forward with MISP/PyMISP #689 Add new feature to delete list of attributes by ID
• Create some other server side methods that can be exposed to the WebUI and PyMISP, e.g. define a list of events and a date range and delete all soft deleted attributes in that data set.

Thanks for your consideration.

[chrisinmtown]

Maybe the scope could be broadened slightly? If a new server-side method supports batch/bulk delete of attributes (note plural), whether soft-deleted or not, that would nicely support this PyMISP feature request: https://github.com/MISP/PyMISP/issues/689

[github-germ]

Is there agreement to offer a server-side attribute batch list deletion of attributes already with soft or hard delete true, and making that available in PyMISP?

[github-germ]

Any new thoughts on this would be appreciated.

[certbe-trey]

bump

[github-germ]

@iglocska This feature request seems to have fallen into a deep hole. Any chance of revisiting this to include soon? Is it really a lot of work to implement? Thanks...

[pietrogu]

Hi,

I would also be interested in this feature

@iglocska @righel what do you think about @github-germ suggestion?

[github-germ]

I see this feature request related:

6933 Ability to Define Hard or Soft Delete for Feed Delta Merge Processing

[github-germ]

Happy Holidays @iglocska + @Rafiot !!

(Hmmm, probably should've posted this to https://github.com/MISP/PyMISP/issues/689)

Any chance of this oldie but goodie getting implemented to offer a client method to delete attributes in bulk? As is now, hard deletes do NOT scale!

Could be as suggested with an PyMISP atomic call with a list of id or uuid.

Or perhaps something similar to how tags can be deleted in a MISPEvent passed to update_event where deleted=1 is set -- I guess we'd also need something like hard=1 for attributes as setting just deleted now does work but only for soft delete when passed in update_event.

THANKS!!

[github-germ]

@iglocska This is an important use case for us. Let us know if we can provide any addition info to help move this feature request forward. Thanks.

[github-germ]

@iglocska Hi... revisiting this with 2.4.191 -- any chance that we can see this feature sometime soon?

[Sp-TT]

looking for this feature as well. We pulling feed to a fixed event. But the soft-deleted attribute list is getting longer and longer

[D3f4ult8]

It would be great to have this feature. At the moment we are pulling some feeds which are dynamically updated every hour and they are generating a lot of soft deleted attributes which we would like to hard delete in an easy way. At the moment we have to manually delete them on a daily basis by filtering for deleted items -> show all -> select all -> delete.

[github-germ]

Revisiting this 3+ year old issue now that 2.4.196 has been released. Are there any plans to implement, and if so, any targeted timeframe? Thanks!