Bug: Live dashboard - attribute with tags

Nicolas-Pellletier commented 1 year ago

Hello,

Steps to reproduce

1°) Have a zmq dashboard that works with ./diagnostic.pynot showing error. (Help you with \logs\*.log files and ) 2°) Add the CERT-FR MISP feed to the misp instance. Then fetch/pull misp events from this source (only 13 events pulled). 3°) ./clean.py for clearing the misp dashboard to remove any data that could come from the fetch. 4°) Go to the "[CERT-FR] Le groupe cybercriminel Silence" event then click on the left side panel Publish event to zmq.

Actual behavior

On the dashboard i see only one attribute integrated. And in the browser console, i see a long list of this error: The error is located here line 345 of the misp-dashboard/static/js/index.js (file):

this.origTableOptions = {
                dom: "<'row'<'col-sm-12'<'dt-toolbar-led'>>>"
                        + "<'row'<'col-sm-12'tr>>",
                searching: false,
                paging:         false,
                "order": [[ 0, "desc" ]],
                responsive: true,
                columnDefs: [
                    { targets: 0, orderable: false },
                    { targets: '_all', searchable: false, orderable: false,
                        render: function ( data, type, row ) {
                            var $toRet;
                            if (typeof data === 'object') {
                                $toRet = $('<span></span>');
                                data.data.forEach(function(cur, i) {
                                    switch (data.name) {
                                        case 'Tag':
                                            var $tag = $('<a></a>');
                                            $tag.addClass('tagElem');
                                            $tag.css({
                                                backgroundColor: cur.colour,
                                                color: getTextColour(cur.colour.substring(1,6)) <-- error happend here
                                            });

Debug I tried to find where the error comes from then i found that: data is an object of this form: console.dir(data) --> Object { name: "Tag", data: Array [ "[object Object]" ] } Object.keys(data.data[0]) --> [ "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", … , "14] then with this function:

function showProps(obj, objName) {
  let result = "";
  for (const i in obj) {
    // Object.hasOwn() is used to exclude properties from the object's
    // prototype chain and only show "own properties"
    if (Object.hasOwn(obj, i)) {
      result += `${objName}.${i} = ${obj[i]}\n`;
    }
  }
  console.log(result);
}

I've got this result for data.data[0].keys = value:

data.0 = [
data.1 = o
data.2 = b
data.3 = j
data.4 = e
data.5 = c
data.6 = t
data.7 =  
data.8 = O
data.9 = b
data.10 = j
data.11 = e
data.12 = c
data.13 = t
data.14 = ]

But i don't know where data.data[0] object is initialized

Nicolas-Pellletier commented 1 year ago

For test you could as well create an event with a single attribute that has a tag attache to it. It's not integrated in the live dashboard due to the error above as well

Nicolas-Pellletier commented 1 year ago

Some update in the debug process The error mentioned is due to an error when an EventMessage object is created in server.py. This object creation happend when logs() function in server.py (line 475) is called. This function is in charge of filling the url https://misp_url:8001/_logs which contains all the data attributes / object attributes sent by the dispatcher.py to the server.py. This data is then used to fill the Logs panel of the Misp Live Dashboard.
The problem is that the Tag information self.feed[2]['data'] of the data send to the live dashboard (see https://misp_url:8001/_logs) is an array of object(s): See below the attributes information sent to the dashboard

self.feed:{0: '2023-03-08 14:15:36', 1: '1635', 2: {'name': 'Tag', 'data': [{'id': '440', 'name': ' Cobalt Strike Beacon', 'colour': '#3b9989', 'exportable': True, 'user_id': '0', 'hide_tag': False, 'numerical_value': None, 'is_galaxy': False, 'is_custom_galaxy': False, 'local_only': False, 'local': 1, 'relationship_type': None}]}, 3: 'Network activity', 4: 'ip-src', 5: '5.39.30.110||'}

Or self.feed[2]['data'] here is an array of object(s). So it's interpreted by the Javascripr as: Array [ "[object Object]" ] (see console.dir(data) above in the Debug part of the first message)

So what we want is a string that we can parse with JSON.parse(tag_info_str) in misp-dashboard/static/js/index/index.js in order to retrieve the tag object. By adding the 2 following lines at l330:

class EventMessage():
    # Suppose the event message is a json with the format {name: 'feedName', log:'logData'}
    def __init__(self, msg, filters):
        if not isinstance(msg, dict):
            try:
                jsonMsg = json.loads(msg)
                jsonMsg['log'] = json.loads(jsonMsg['log'])
            except json.JSONDecodeError as e:
                logger.error(e)
                jsonMsg = { 'name': "undefined" ,'log': json.loads(msg) }
        else:
            jsonMsg = msg

        self.name = jsonMsg['name']
        self.zmqName = jsonMsg['zmqName']

        if self.name == 'Attribute':
            self.feed = jsonMsg['log']
            self.feed = LogItem(self.feed, filters).get_row()
           #Here self.feed which is jsonMsg['log'] 
           if (self.feed[2]):                                       <--- Check if msg received from the dispatcher contains a Tag field in attributes infos
            self.feed[2]['data'] = json.dumps(self.feed[2]['data']) <--- Transform the attribute's tag array in array of strings instead of array of object (which was causing the bug)

        elif self.name == 'ObjectAttribute':
            self.feed = jsonMsg['log']
            self.feed = LogItem(self.feed, filters).get_row()
        else:
            self.feed = jsonMsg['log']
            ....

We get: this result:

self.feed:{0: '2023-03-08 14:15:36', 1: '1635', 2: {'name': 'Tag', 'data': '[{"id": "440", "name": " Cobalt Strike Beacon", "colour": "#3b9989", "exportable": true, "user_id": "0", "hide_tag": false, "numerical_value": null, "is_galaxy": false, "is_custom_galaxy": false, "local_only": false, "local": 1, "relationship_type": null}]'}, 3: 'Network activity', 4: 'ip-src', 5: '5.39.30.110||'}

Then adding data.data = JSON.parse(data.data); at line 337 in static/js/index/index.js where the data is processed to get the jquery column parts. We can see by posing breakpoint at the same line and stepping through this render function that we got a proper HTML render for the Attrbute.Tag cells value: <span><a class="tagElem" style="background-color: rgb(59, 153, 137); color: rgb(255, 255, 255);"> Cobalt Strike Beacon</a></span>

See Below full render function of the index.js l.323. I've put some comment (and code for debug purpose) in the code below in order to help you debug and see what the code do...

// create table and draw header
            this.origTableOptions = {
                dom: "<'row'<'col-sm-12'<'dt-toolbar-led'>>>"
                        + "<'row'<'col-sm-12'tr>>",
                searching: false,
                paging:         false,
                "order": [[ 0, "desc" ]],
                responsive: true,
                columnDefs: [
                    { targets: 0, orderable: false },
                    { targets: '_all', searchable: false, orderable: false,
                        render: function ( data, type, row ) {
                            var $toRet;
                            data.data = JSON.parse(data.data); // <--- Here we retrieve the Tag Object                          
                            console.log("data:"+ data + "\ntype:\"" + type + "\"\nrow:\"" + JSON.stringify(row, null, 2) + "\n");
                           //For debug purpose break here and look at the console to see (data | type | row) values
                            if (typeof data === 'object') {
                                //Here we are in Tag object 
                                $toRet = $('<span></span>');
                                data.data.forEach(function(cur, i) {
                                    switch (data.name) {
                                        case 'Tag':
                                            var $tag = $('<a></a>');
                                            $tag.addClass('tagElem');
                                            $tag.css({
                                                backgroundColor: cur.colour,
                                                color: getTextColour(cur.colour.substring(1,6))
                                            });
                                            $tag.text(cur.name)
                                            $toRet.append($tag);
                                            break;
                                        case 'mispObject':
                                            $toRet.append('MISP Object not supported yet')
                                            break;
                                        default:
                                            break;
                                    }
                                });
                                $toRet = $toRet[0].outerHTML;
                            } else if (data === undefined) {
                                    $toRet = '';
                            } else {
                                var textToAddArray = data.split(char_separator);
                                $toRet = '';
                                textToAddArray.forEach(function(e, i) {
                                    if (i > 0) {
                                        $toRet += '<br>' + e;
                                    } else {
                                        $toRet += e;
                                    }
                                });
                            }
                            //Break here to see $toRet HTML value of cell of the  Attribute.Tag column
                            console.log($toRet);  
                            return $toRet;
                       },
                    }
                ],
            };

Nicolas-Pellletier commented 1 year ago

ISSUE FIXED Then i finally resolve the issue and got a tag in the Attribute.Tag column for Attribute(s) or Object Attribute(s) that got a tag attach to them. See below the picture:

I will open a pull request but in case it's not accepted, you just have to do few changes: 1) In misp-dashboard/server.py file line 328 just replace this:

        if self.name == 'Attribute':
            self.feed = jsonMsg['log']
            self.feed = LogItem(self.feed, filters).get_row()
        elif self.name == 'ObjectAttribute':
            self.feed = jsonMsg['log']
            self.feed = LogItem(self.feed, filters).get_row()
        else:
            self.feed = jsonMsg['log']

by this:


        if self.name == 'Attribute':
            self.feed = jsonMsg['log']
            self.feed = LogItem(self.feed, filters).get_row()
            if (self.feed[2]):
                 self.feed[2]['data'] = json.dumps(self.feed[2]['data'])
        elif self.name == 'ObjectAttribute':
            self.feed = jsonMsg['log']
            self.feed = LogItem(self.feed, filters).get_row()
            if (self.feed[2]):
                 self.feed[2]['data'] = json.dumps(self.feed[2]['data'])
        else:
            self.feed = jsonMsg['log']

2) In misp-dashboard/static/js/index/index.js file line 337 just replace this:

 render: function ( data, type, row ) {
                            var $toRet;
                            if (typeof data === 'object') {
                                $toRet = $('<span></span>');
                                data.data.forEach(function(cur, i) {
                                    switch (data.name) {
                                        case 'Tag':

By this:

render: function ( data, type, row ) {
                            var $toRet;
                            if (typeof data === 'object') {
                                $toRet = $('<span></span>');
                                let tagList = JSON.parse(data.data);
                                tagList.forEach(function(cur, i) {
                                    switch (data.name) {
                                        case 'Tag':

MISP / misp-dashboard

Bug: Live dashboard - attribute with tags #171