search

certtools / intelmq

IntelMQ is a solution for IT security teams for collecting and processing security feeds using a message queuing protocol.
https://docs.intelmq.org/latest/
GNU Affero General Public License v3.0
972 stars 295 forks source link

Misp Collector/parser refactoring #1422

Open kalyparker opened 5 years ago

kalyparker commented 5 years ago

Hello,

I'm working on the Collector/parser bot for misp. There are some things which I don't like and want to change.

So before doing it, maybe you have additional idea :)

Basically, I change the mandatory tag rewriting. This option should be optional, when you used a shared instance you cannot update events you not own.

Regarding the filter, I will plan to add: date => I wish to filter on the last published event. Relative time: 2d, 1h... company => I'm not sure to success with this one, idea should be to grab events coming from a particular instance (synchronisation with external misp - called server in misp) eventid => I know this information depend on the instance, so this field is optional

Regarding information, like asking in issue #1357, I'll add event name. Any idea where to put this information ? I used the field "Comment", maybe a new one can be created in the harmonization?

Bonus: I wonder if it is possible to create a "matrix like" for matching classification with a configuration file. Misp is flexible, maybe too much: https://github.com/MISP/misp-taxonomies

Actually I grab the malware name, based on the parsing of event tag, which is not great, but often works. It is complicated due to the taxonomies jungle. For the classification, I wish to grab minimal information => is it a phishing, a c2 ... Again, it is probably on the event name or in the tag. Do you have any idea for achieving this?

Last question: I plan to use pymisp, is it a problem?

navtej commented 5 years ago

I had run into problems with MISP. MISP search api does not support pagination of results. For each query it would find all result and send you. The query does couple of SQL joins and they can bring a small instance down fairly quickly. If MISP has solved the underlying problem, it would be helpful to integrate that pagination with IMQ.

Another improvement we can do is wrt to feed url and name. As of today IMQ use the feed.url as MISP instance url. A flag can support using either generic url and name or the name and url available from the MISP event.

ghost commented 5 years ago

cc @Rafiot @kralca (contributors/authors of both bots)

Rafiot commented 5 years ago

@navtej the search API now supports pagination (with the limit and page parameters), so that will most probably help. I'm not an IntelMQ user, so I cannot give much more details, but I agree with @kalyparker approach.

One important thing tho: I'm currently preparing a clean move to python 3.6 and deprecate everything that will go away early 2020. If you exclusively use ExpandedPyMISP, you are safe, but if you give me a few more days, the refactoring will be done.

ghost commented 4 years ago

May it make sense to collect data using the subscribe possibility of misp zmq? See https://www.circl.lu/doc/misp/misp-zmq/