Embedded backslashes in attribute value vanish after adding to event

[chrisinmtown]

How can I preserve backslashes in attribute values when I add an attribute to an existing event? The backslashes don't make it to the database. I don't see any messages in debug.log/error.log, and the PyMISP.add_attribute() invocation returns success. I'm using PyMISP 2.4.137 and the code basically is this:

    attr = pymisp.MISPAttribute()
    attr.type = aType
    attr.value = aValue
    attr.category = aCategory
    r = misp.add_attribute(event['Event']['id'], attr)

Here is the value that I read from an external source:

http://googlechromeupdater.twilightparadox.com/html?DVXNSTHORF=fd6f240590734406be3bd35ca3622ea0;GRIBOOZ0LN=a3bf23855b0b40dda08f709fabb60d32;\..\..\..\./mshtml,RunHTMLApplication

The value stored in the database has no backslashes:

mysql> select * from attributes where id = 154926899 \G
*************************** 1. row ***************************
                 id: 154926899
           event_id: 193076
          object_id: 0
    object_relation: NULL
           category: Network activity
               type: url
             value1: http://googlechromeupdater.twilightparadox.com/html?DVXNSTHORF=fd6f240590734406be3bd35ca3622ea0;GRIBOOZ0LN=a3bf23855b0b40dda08f709fabb60d32;......./mshtml,RunHTMLApplication
             value2: 
             to_ids: 1
               uuid: 8edb38f3-e3be-4f76-ac0f-784c154a3756
          timestamp: 1615908632
       distribution: 5
   sharing_group_id: 0
            comment: 
            deleted: 0
disable_correlation: 0
         first_seen: NULL
          last_seen: NULL
1 row in set (0.00 sec)

[Rafiot]

I'm guessing it's stripped on MISP side? There is no processing on PyMISP side for the values. @iglocska @mokaddem, do you know what's going on there?

[chrisinmtown]

Here's code you can run for yourself to see it.

#!/usr/bin/env python3

import sys
import urllib3
from pymisp import PyMISP, MISPAttribute, MISPEvent
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

def check_misp_response(resp: dict, msg: str):
    """Check a MISP response for error; show error and exit if found"""
    if isinstance(resp, dict) and 'errors' in resp:
        print('{} failed: {}'.format(msg, resp['errors']), file=sys.stderr)
        raise Exception('check_misp_response exiting')

misp_client = PyMISP(
    url='https://misp.mycompany.com:48752',
    key='key',
    ssl=False)
if not misp_client:
    print('PyMISP() failed\n', file=sys.stderr)
    sys.exit(1)

event = MISPEvent()
event.distribution = 0  # internal
event.published = False
event.info = 'Test event for attribute values with backslashes'
event = misp_client.add_event(event, pythonify=True)
check_misp_response(event, 'add_event')
print('created event:\n{}'.format(event.to_json(sort_keys=True, indent=4)))
attr = MISPAttribute()
attr.category = 'Network activity'
attr.type = 'url'
attr.value = r'http://forward.com/foo\.\.\.'
attr = misp_client.add_attribute(event, attr, pythonify=True)
check_misp_response(attr, 'add_attribute')
print('created attr:\n{}'.format(attr.to_json(sort_keys=True, indent=4)))
response = misp_client.delete_event(event)
check_misp_response(attr, 'delete_event')

Here's output on my site, a couple strings fiddled to protect the innocent lol:

% python3 create_attr_backsl.py
created event:
{
    "Org": {
        "id": "2",
        "local": true,
        "name": "ABC",
        "uuid": "5a6a4161-b2e0-46b6-b12e-1bca0a6b037c"
    },
    "Orgc": {
        "id": "2",
        "local": true,
        "name": "ABC",
        "uuid": "5a6a4161-b2e0-46b6-b12e-1bca0a6b037c"
    },
    "analysis": "0",
    "attribute_count": "0",
    "date": "2021-03-16",
    "disable_correlation": false,
    "distribution": "0",
    "event_creator_email": "me@mycompany.com",
    "extends_uuid": "",
    "id": "223510",
    "info": "Test event for attribute values with backslashes",
    "locked": false,
    "org_id": "2",
    "orgc_id": "2",
    "proposal_email_lock": false,
    "publish_timestamp": 0,
    "published": false,
    "sharing_group_id": "0",
    "threat_level_id": "4",
    "timestamp": "1615921539",
    "uuid": "052c2bc9-5c92-446c-a94c-95a510b71345"
}
created attr:
{
    "category": "Network activity",
    "comment": "",
    "deleted": false,
    "disable_correlation": false,
    "distribution": "5",
    "event_id": "223510",
    "id": "154932165",
    "object_id": "0",
    "sharing_group_id": "0",
    "timestamp": "1615921539",
    "to_ids": true,
    "type": "url",
    "uuid": "0b774606-3012-4851-a5bf-b4c6c41b25c6",
    "value": "http://forward.com/foo...",
    "value1": "http://forward.com/foo...",
    "value2": ""
}

[Rafiot]

A couple of tests:

In [7]: event = MISPEvent()                                                                                                                                                                                      

In [8]: event.info = 'blah'                                                                                                                                                                                      

In [9]: event.add_attribute('url', r'http://forward.com/foo\.\.\.')                                                                                                                                              
Out[9]: <MISPAttribute(type=url, value=http://forward.com/foo\.\.\.)

In [10]: event.to_json()                                                                                                                                                                                         
Out[10]: '{"uuid": "ed151749-1dd5-4767-9357-3802aec71c10", "Attribute": [{"uuid": "7cc45504-6220-4d24-8b69-82967985d8ff", "type": "url", "value": "http://forward.com/foo\\\\.\\\\.\\\\.", "category": "Network activity", "to_ids": true, "disable_correlation": false}], "info": "blah"}'

In [11]: e = misp.add_event(event, pythonify=True)

In [12]: e.to_json()                                                                                                                                                                                             
Out[12]: '{"uuid": "ed151749-1dd5-4767-9357-3802aec71c10", "Attribute": [{"uuid": "7cc45504-6220-4d24-8b69-82967985d8ff", "type": "url", "value": "http://forward.com/foo...", "category": "Network activity", "to_ids": true, "distribution": "5", "id": "1", "event_id": "1", "timestamp": "1615922178", "sharing_group_id": "0", "disable_correlation": false, "comment": "", "deleted": false, "object_id": "0"}], "info": "blah", "distribution": "1", "threat_level_id": "4", "analysis": "0", "published": false, "date": "2021-03-16", "id": "1", "orgc_id": "1", "org_id": "1", "timestamp": "1615922178", "publish_timestamp": 0, "sharing_group_id": "0", "Org": {"id": "1", "name": "ORGNAME", "uuid": "5d397c4b-3474-466a-80cd-624838d4ff94", "local": true}, "Orgc": {"id": "1", "name": "ORGNAME", "uuid": "5d397c4b-3474-466a-80cd-624838d4ff94", "local": true}, "attribute_count": "1", "proposal_email_lock": false, "locked": false, "disable_correlation": false, "extends_uuid": "", "event_creator_email": "admin@admin.test"}'

So yeah, the attribute sent has the \. and they are stripped on MISP side.

[chrisinmtown]

Thanks @Rafiot should I move this issue to the MISP repo, close it in PyMISP?

[Rafiot]

So I'm not sure if it is a bug: based on the 5 min search I made, the \ don't seem to be supported by many URL processors. It seems to be completely striped and ignored most of the time.

So yeah, move it to MISP, but please also explain where the URL comes from, and what is the usecase to have such a string in a URL field.

[chrisinmtown]

The usecase is simple - this is an IOC sent by a premium supplier, presumably after observing it in an attack. I don't know the meaning. I just want MISP to store & display it faithfully.

[chrisinmtown]

Opened issue https://github.com/MISP/MISP/issues/7214

[chrisinmtown]

Hmm the MISP server crowd seems to think this happens when PyMISP serializes my data into JSON. Perhaps this is a PyMISP issue after all, and the client code should be calling Python's ".encode()" method on strings?

I'm taking the liberty of reopening this issue. If that's wrong, please guide me.

[iglocska]

I was wrong, this is indeed a MISP side issue - should be fixed as of a few minutes ago. Well spotted!