CEF Export Example

[Amorik]

Can Some please provide an example of how to use the CEF Export module, or any export module for that fact. The documentation is poor in this regard and it is not clear on how to actually use most of the modules.

Thank you

~Regards

[frennkie]

Crazy.. I was just now trying to figure out the exact same thing..

This guys seems to be confused too: #54

I didn't find out how to test on the CLI (e.g. with curl) but when you enable an export module (e.g. set Plugin.Export_cef_export_enabled to true) in the admin page (Administration-> Server settings -> Plugin settings -> Export) then you can open any Event that you have in MISP and click on Download as... on the left panel.. there you have access to e.g. Cef Export. This will export the attributes of the Event in the selected format.

[Amorik]

Didn't know the option was there. Looking at the download url seems like the missing link was the sub url of events. Using http:<yourmispserver>/events/exportModule/cef_export does the trick for all events (though only those that have attributes per the coded mapping - an option to be able to alter this would be nice)

Still working out the CLI but it seems you pass the event results from the json format as the "data" attribute. Short Example: curl -s http://<yourmispmoduleserver>:<yourmispmoduleserverport>/query -H "Content-Type: application/json" -d '{ "module": "cef_export", "data": [ <json formatted list of events> ] }'

The module loops through the events and loops through the "Attributes" within each event to get the output. It might be possible to edit the mapping in the request, haven't tried yet.

[TostyT]

Any luck with configuring an export module? I just came across this issue and was wondering if you guys made any progress?

[Amorik]

If were talking about the CEF module, installed with the misp-modules package, you will find settings under the "Server Settings & Maintenance > Plugin Settings > Export" section.

As long as "Plugin.Export_cef_export_enabled" is set to true you will be able to use the CEF export option when viewing Events. This only works for the "Download as..." option when viewing events and not in the export section where one would grab say a CSV, Bro, XML, ect. You can export all events in CEF using the method

There aren't many options to customize the output and you may or not agree with what is extracted from the MISP information. You can adjust this, but requires you to modify the module code directly, or write your own. https://www.circl.lu/assets/files/misp-training/brussels2016/misp-modules.pdf is a great source to get started, that and use the exiting CEF module to learn from goes a long way. You can easily copy the code, and rename it. For example I've copied the "cef_export.py" file to "my_cef_export.py" and modified lines 38 through 48

From:

          response += "{} host CEF:0|{}|{}|{}|{}|{}|{}|{}={}\n".format(
                            datetime.datetime.fromtimestamp(int(attr["timestamp"])).strftime("%b %d %H:%M:%S"),
                            config["Device_Vendor"],
                            config["Device_Product"],
                            config["Device_Version"],
                            attr["category"],
                            attr["category"],
                            config["Default_Severity"],
                            cefmapping[attr["type"]],
                            attr["value"],
                    )

To:

          response += "CEF:0|{}|{}|{}|{}|{}|{}|{}={} endTime={} dvcHost={} cs1Label=Event Org cs1={} cs2Label=Event UUID cs2={} cs3Label=Attribute Org cs3='' cs4Label=Attribute UUID cs4={} \n".format(
                            config["Device_Vendor"],
                            config["Device_Product"],
                            config["Device_Version"],
                            attr["category"],
                            attr["category"],
                            config["Default_Severity"],
                            cefmapping[attr["type"]],
                            attr["value"],
                            datetime.datetime.fromtimestamp(int(attr["timestamp"])).timestamp(),
                            socket.gethostname(),
                            ev.keys(),
                            ev.keys(),
                            attr["uuid"],
                    )

Then restarted the misp-modules process. Monitoring the log files for issues and resolve as necessary. Once its working you should see the new export module in the server settings and in the Download as.. options. This allowed me to add several new static fields based on the event and include additional specific MISP attribute data. You will have to play with the output to learn what can be done; note adding print statements should output to the logfile.

As for using the API to trigger the export, outside what I've previously posted, i have not been able to adjust it any further. At present I pull and process events in NiFI, but one could also do the same continuing the PyMISP method; though for now i am not show how to make it more of a passive module like say ZeroMQ

Hope that helps.

~Regards