CERT-Polska / mwdb-core

Malware repository component for samples & static configuration with REST API interface.
https://mwdb.readthedocs.io/
Other
323 stars 74 forks source link

Support user-defined queries for easy access #33

Closed ITAYC0HEN closed 4 years ago

ITAYC0HEN commented 4 years ago

Describe the problem you are facing Currently, when I want to execute the same set of queries in mwdb frequently (once a day, once a week, ...) I don't have any easy way to do this. Only by typing them ad-hoc or keeping a list of queries at the side.

Describe the solution you'd expect I would like to have an option to save queries (e.g tag:"yara:win_formbook" AND NOT tag:"ripped:*") so I can quickly access them when I need. In this solution, I could visit the Search page, or a special dashboard, and choose a query I want to execute, without having to type it manually or pasting it from notepad.

msm-code commented 4 years ago

That's a pretty interesting idea. This topic was also raised frequently, but became less hot internally after the quick query feature (it's like saved searches but hardcoded to 4 common cases).

Food for thought: what about sharing your saved searches with your groups/organisations?

I remember @psrok1 thought about it, so let's wait for him to decide.

ITAYC0HEN commented 4 years ago

Bonus points for a little badge near each query with number of notifications didn't read. This will allow us to "hunt" for interesting cases using mwdb, whether it's new variants, configuration change, unique collaboration between multiple malware families and more :)

Say, so here are my 7 saved queries, 3 of them have new results since I last watched them.

For a very noisy query, the interface can show "100+"and that's it, without having to trigger the query again until this goes under 100 (for example)

msm-code commented 4 years ago

Sounds nice! Though it doubles the implementation complexity, so we should start with a simple version (just save searches, without notifications).

But that's certainly doable and a nice idea.

psrok1 commented 4 years ago

Sounds cool 👍 I think this feature can be also easily extended by "catalogue of queries" (Stored queries) with user-defined queries that are:

Queries can use specialized fields depending on type and this needs to be considered during implementation. For example: file.md5:... used on Recent configs view will throw an error Can't search for objects with type 'File' and 'Config' in the same query. Simple md5:... will report No such field: md5, because configs doesn't have such field.

Notifications part would be much more complicated, mainly due to the cost of querying database and possible condition complexity. We could limit this a bit by:

Then we can just lazy-load these "counters" by executing all notification-enabled queries. Storing them in database and keeping them up-to-date sounds like lots of things that can go wrong.

psrok1 commented 4 years ago

Implementation limited to pinnable queries in Quick query bar:

image

ITAYC0HEN commented 2 years ago

@psrok1 @msm-code @nazywam I think we can re-surface the Notifications discussion again. I find it very useful to be able to get notifications for saved queries.

What do you think?