IntelMQ and MongoDB

[Flo1895]

I started to work with IntelMQ a few days ago and for testing purposes I wanted to create a little pipeline which collects a few threat intelligence feeds and stores them in a MongoDB database. The parser bots create json documents according to the harmonization.conf file. However, a lot of field names contain one or more dots, which is not allowed when storing json documents in a MongoDB database. Are you aware of that issue, and is there any way of bypassing that? Otherwise, the MongoDB output bot is rather useless in my opinion.

[dmth]

Hi, thank you for your report.

Maybe you can try something: Does it help to set the MongoDB outputbots hierarchical_output parameter to True in your case?

[Flo1895]

Hi, thank you for your fast reply. Setting hierarchical_output to True indeed fixed my problem so far. However, I am not sure if it may lead to "problems" or rather inconsistencies when working with the data from the mongodb. The document {"classification.identifier": "bla"} will now be inserted into the mongodb as {"classification": {"identifier": "bla"}} which contradicts in a way the definition in harmonization.conf. Of course, I could chain classification and identifier back together after retrieving the data from MongoDB, but it would take some effort and you have to be very careful to chain only together what belongs together.

[sebix]

If Mongodb does not support dots in keys, there's nothing we can do about it.