search

MISP / PyMISP

Python library using the MISP Rest API
Other
444 stars 279 forks source link

Adding custom object to event #462

Closed thomasb454 closed 4 years ago

thomasb454 commented 5 years ago

Hi,

I'm having issues when trying to update an event with a custom object.

PyMISP version: 2.4.114 Python version: 3.7

The object template is present on the web server (I can manually add the object to an event), however when using ExpandedPyMISP it returns a 403. ERROR [aping.py:2039 - _check_response() ] Something went wrong (403): {'name': 'Could not add object', 'message': 'Could not add object', 'url': '/objects/add/9/', 'errors': 'No valid template found to edit the object.'}

from pymisp import ExpandedPyMISP
from pymisp import MISPObject

misp = ExpandedPyMISP('<url>', '<key>', False)

event = misp.search(uuid='5d89ea9a-0778-4a90-a4d0-05a7ac1002c7')

obj = MISPObject(name='mopr-report', misp_objects_path_custom='mopr')

obj.add_attribute('score', value=0.02)
obj.add_attribute('malicious', value=False)
obj.add_attribute('benign', value=True)

misp.add_object('5d89ea9a-0778-4a90-a4d0-05a7ac1002c7', misp_object=obj)

Any advice is appreciated.

Rafiot commented 5 years ago

The problem is that MISP

  1. doesn't know the object template
  2. tries to update an existing event on the platform

when the two conditions are true, the error is expected.

You can either add the template on MISP, or create a new object (by changing the UUID).

thomasb454 commented 5 years ago

Hi,

The object template is present on MISP? Like I said, I can add the object to an event using the web app. Additionally, you can see I point the MISPObject to the custom template locally (so it can load all the related metadata and perform validiation)

Rafiot commented 5 years ago

Yes, you can add it, but you should not be able to update it from the web interface (?)

If you can, I'm confused and will ask @iglocska to the rescue.

thomasb454 commented 5 years ago

Hi,

I can add the object to an event and edit it (on the web app).

Okay haha - thank you.

thomasb454 commented 5 years ago

Hi,

I've found a less than ideal workaround. Hopefully the above issue is fixed because these methods are deprecated.

Instead of using the new ExpandedPyMISP API, revert to PyMISP API and provide the template UUID when adding the object to the event.

kovacsbalu commented 5 years ago

Hi @thomasb454, I tried to reproduce you issue. There is no error for me. My MISP version is 2.4.109 pymisp 2.4.114 In your error msg there is an 'url': '/objects/add/9/'. Object index 9 is you custom object id? Did you try debug mode?

thomasb454 commented 5 years ago

Hi @kovacsbalu,

9 is the ID of the event that I'm trying to add the object to. Debug output produces the following:

DEBUG [aping.py:2073 - _prepare_request() ] GET - http://<url>/servers/getPyMISPVersion.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.7', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'm3uBBfGH94cDSSsDvZv7Doqpihgsjg1Zc3WPYs5r', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'version': '2.4.99'}
INFO [aping.py:79 - __init__() ] The version of PyMISP recommended by the MISP instance (response['version']) is older than the one you're using now (2.4.114). If you have a problem, please upgrade the MISP instance or use an older PyMISP version.
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://<url>/servers/getVersion.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.7', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'm3uBBfGH94cDSSsDvZv7Doqpihgsjg1Zc3WPYs5r', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'version': '2.4.100', 'perm_sync': True}
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://<url>/attributes/describeTypes.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.7', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'm3uBBfGH94cDSSsDvZv7Doqpihgsjg1Zc3WPYs5r', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'result': {'sane_defaults': {'md5': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha1': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha256': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename': {'default_category': 'Payload delivery', 'to_ids': 1}, 'pdb': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'filename|md5': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha1': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha256': {'default_category': 'Payload delivery', 'to_ids': 1}, 'ip-src': {'default_category': 'Network activity', 'to_ids': 1}, 'ip-dst': {'default_category': 'Network activity', 'to_ids': 1}, 'hostname': {'default_category': 'Network activity', 'to_ids': 1}, 'domain': {'default_category': 'Network activity', 'to_ids': 1}, 'domain|ip': {'default_category': 'Network activity', 'to_ids': 1}, 'email-src': {'default_category': 'Payload delivery', 'to_ids': 1}, 'email-dst': {'default_category': 'Network activity', 'to_ids': 1}, 'email-subject': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-attachment': {'default_category': 'Payload delivery', 'to_ids': 1}, 'email-body': {'default_category': 'Payload delivery', 'to_ids': 0}, 'float': {'default_category': 'Other', 'to_ids': 0}, 'url': {'default_category': 'Network activity', 'to_ids': 1}, 'http-method': {'default_category': 'Network activity', 'to_ids': 0}, 'user-agent': {'default_category': 'Network activity', 'to_ids': 0}, 'ja3-fingerprint-md5': {'default_category': 'Network activity', 'to_ids': 1}, 'regkey': {'default_category': 'Persistence mechanism', 'to_ids': 1}, 'regkey|value': {'default_category': 'Persistence mechanism', 'to_ids': 1}, 'AS': {'default_category': 'Network activity', 'to_ids': 0}, 'snort': {'default_category': 'Network activity', 'to_ids': 1}, 'bro': {'default_category': 'Network activity', 'to_ids': 1}, 'pattern-in-file': {'default_category': 'Payload installation', 'to_ids': 1}, 'pattern-in-traffic': {'default_category': 'Network activity', 'to_ids': 1}, 'pattern-in-memory': {'default_category': 'Payload installation', 'to_ids': 1}, 'yara': {'default_category': 'Payload installation', 'to_ids': 1}, 'stix2-pattern': {'default_category': 'Payload installation', 'to_ids': 1}, 'sigma': {'default_category': 'Payload installation', 'to_ids': 1}, 'gene': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'mime-type': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'identity-card-number': {'default_category': 'Person', 'to_ids': 0}, 'cookie': {'default_category': 'Network activity', 'to_ids': 0}, 'vulnerability': {'default_category': 'External analysis', 'to_ids': 0}, 'attachment': {'default_category': 'External analysis', 'to_ids': 0}, 'malware-sample': {'default_category': 'Payload delivery', 'to_ids': 1}, 'link': {'default_category': 'External analysis', 'to_ids': 0}, 'comment': {'default_category': 'Other', 'to_ids': 0}, 'text': {'default_category': 'Other', 'to_ids': 0}, 'hex': {'default_category': 'Other', 'to_ids': 0}, 'other': {'default_category': 'Other', 'to_ids': 0}, 'named pipe': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'mutex': {'default_category': 'Artifacts dropped', 'to_ids': 1}, 'target-user': {'default_category': 'Targeting data', 'to_ids': 0}, 'target-email': {'default_category': 'Targeting data', 'to_ids': 0}, 'target-machine': {'default_category': 'Targeting data', 'to_ids': 0}, 'target-org': {'default_category': 'Targeting data', 'to_ids': 0}, 'target-location': {'default_category': 'Targeting data', 'to_ids': 0}, 'target-external': {'default_category': 'Targeting data', 'to_ids': 0}, 'btc': {'default_category': 'Financial fraud', 'to_ids': 1}, 'xmr': {'default_category': 'Financial fraud', 'to_ids': 1}, 'iban': {'default_category': 'Financial fraud', 'to_ids': 1}, 'bic': {'default_category': 'Financial fraud', 'to_ids': 1}, 'bank-account-nr': {'default_category': 'Financial fraud', 'to_ids': 1}, 'aba-rtn': {'default_category': 'Financial fraud', 'to_ids': 1}, 'bin': {'default_category': 'Financial fraud', 'to_ids': 1}, 'cc-number': {'default_category': 'Financial fraud', 'to_ids': 1}, 'prtn': {'default_category': 'Financial fraud', 'to_ids': 1}, 'phone-number': {'default_category': 'Person', 'to_ids': 0}, 'threat-actor': {'default_category': 'Attribution', 'to_ids': 0}, 'campaign-name': {'default_category': 'Attribution', 'to_ids': 0}, 'campaign-id': {'default_category': 'Attribution', 'to_ids': 0}, 'malware-type': {'default_category': 'Payload delivery', 'to_ids': 0}, 'uri': {'default_category': 'Network activity', 'to_ids': 1}, 'authentihash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'ssdeep': {'default_category': 'Payload delivery', 'to_ids': 1}, 'imphash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'pehash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'impfuzzy': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha224': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha384': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha512': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha512/224': {'default_category': 'Payload delivery', 'to_ids': 1}, 'sha512/256': {'default_category': 'Payload delivery', 'to_ids': 1}, 'tlsh': {'default_category': 'Payload delivery', 'to_ids': 1}, 'cdhash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|authentihash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|ssdeep': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|imphash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|impfuzzy': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|pehash': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha224': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha384': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha512': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha512/224': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|sha512/256': {'default_category': 'Payload delivery', 'to_ids': 1}, 'filename|tlsh': {'default_category': 'Payload delivery', 'to_ids': 1}, 'windows-scheduled-task': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'windows-service-name': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'windows-service-displayname': {'default_category': 'Artifacts dropped', 'to_ids': 0}, 'whois-registrant-email': {'default_category': 'Attribution', 'to_ids': 0}, 'whois-registrant-phone': {'default_category': 'Attribution', 'to_ids': 0}, 'whois-registrant-name': {'default_category': 'Attribution', 'to_ids': 0}, 'whois-registrant-org': {'default_category': 'Attribution', 'to_ids': 0}, 'whois-registrar': {'default_category': 'Attribution', 'to_ids': 0}, 'whois-creation-date': {'default_category': 'Attribution', 'to_ids': 0}, 'x509-fingerprint-sha1': {'default_category': 'Network activity', 'to_ids': 1}, 'x509-fingerprint-md5': {'default_category': 'Network activity', 'to_ids': 1}, 'x509-fingerprint-sha256': {'default_category': 'Network activity', 'to_ids': 1}, 'dns-soa-email': {'default_category': 'Attribution', 'to_ids': 0}, 'size-in-bytes': {'default_category': 'Other', 'to_ids': 0}, 'counter': {'default_category': 'Other', 'to_ids': 0}, 'datetime': {'default_category': 'Other', 'to_ids': 0}, 'cpe': {'default_category': 'Other', 'to_ids': 0}, 'port': {'default_category': 'Network activity', 'to_ids': 0}, 'ip-dst|port': {'default_category': 'Network activity', 'to_ids': 1}, 'ip-src|port': {'default_category': 'Network activity', 'to_ids': 1}, 'hostname|port': {'default_category': 'Network activity', 'to_ids': 1}, 'mac-address': {'default_category': 'Network activity', 'to_ids': 0}, 'mac-eui-64': {'default_category': 'Network activity', 'to_ids': 0}, 'email-dst-display-name': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-src-display-name': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-header': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-reply-to': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-x-mailer': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-mime-boundary': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-thread-index': {'default_category': 'Payload delivery', 'to_ids': 0}, 'email-message-id': {'default_category': 'Payload delivery', 'to_ids': 0}, 'github-username': {'default_category': 'Social network', 'to_ids': 0}, 'github-repository': {'default_category': 'Social network', 'to_ids': 0}, 'github-organisation': {'default_category': 'Social network', 'to_ids': 0}, 'jabber-id': {'default_category': 'Social network', 'to_ids': 0}, 'twitter-id': {'default_category': 'Social network', 'to_ids': 0}, 'first-name': {'default_category': 'Person', 'to_ids': 0}, 'middle-name': {'default_category': 'Person', 'to_ids': 0}, 'last-name': {'default_category': 'Person', 'to_ids': 0}, 'date-of-birth': {'default_category': 'Person', 'to_ids': 0}, 'place-of-birth': {'default_category': 'Person', 'to_ids': 0}, 'gender': {'default_category': 'Person', 'to_ids': 0}, 'passport-number': {'default_category': 'Person', 'to_ids': 0}, 'passport-country': {'default_category': 'Person', 'to_ids': 0}, 'passport-expiration': {'default_category': 'Person', 'to_ids': 0}, 'redress-number': {'default_category': 'Person', 'to_ids': 0}, 'nationality': {'default_category': 'Person', 'to_ids': 0}, 'visa-number': {'default_category': 'Person', 'to_ids': 0}, 'issue-date-of-the-visa': {'default_category': 'Person', 'to_ids': 0}, 'primary-residence': {'default_category': 'Person', 'to_ids': 0}, 'country-of-residence': {'default_category': 'Person', 'to_ids': 0}, 'special-service-request': {'default_category': 'Person', 'to_ids': 0}, 'frequent-flyer-number': {'default_category': 'Person', 'to_ids': 0}, 'travel-details': {'default_category': 'Person', 'to_ids': 0}, 'payment-details': {'default_category': 'Person', 'to_ids': 0}, 'place-port-of-original-embarkation': {'default_category': 'Person', 'to_ids': 0}, 'place-port-of-clearance': {'default_category': 'Person', 'to_ids': 0}, 'place-port-of-onward-foreign-destination': {'default_category': 'Person', 'to_ids': 0}, 'passenger-name-record-locator-number': {'default_category': 'Person', 'to_ids': 0}, 'mobile-application-id': {'default_category': 'Payload delivery', 'to_ids': 1}, 'cortex': {'default_category': 'External analysis', 'to_ids': 0}, 'boolean': {'default_category': 'Other', 'to_ids': 0}}, 'types': ['md5', 'sha1', 'sha256', 'filename', 'pdb', 'filename|md5', 'filename|sha1', 'filename|sha256', 'ip-src', 'ip-dst', 'hostname', 'domain', 'domain|ip', 'email-src', 'email-dst', 'email-subject', 'email-attachment', 'email-body', 'float', 'url', 'http-method', 'user-agent', 'ja3-fingerprint-md5', 'regkey', 'regkey|value', 'AS', 'snort', 'bro', 'pattern-in-file', 'pattern-in-traffic', 'pattern-in-memory', 'yara', 'stix2-pattern', 'sigma', 'gene', 'mime-type', 'identity-card-number', 'cookie', 'vulnerability', 'attachment', 'malware-sample', 'link', 'comment', 'text', 'hex', 'other', 'named pipe', 'mutex', 'target-user', 'target-email', 'target-machine', 'target-org', 'target-location', 'target-external', 'btc', 'xmr', 'iban', 'bic', 'bank-account-nr', 'aba-rtn', 'bin', 'cc-number', 'prtn', 'phone-number', 'threat-actor', 'campaign-name', 'campaign-id', 'malware-type', 'uri', 'authentihash', 'ssdeep', 'imphash', 'pehash', 'impfuzzy', 'sha224', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'tlsh', 'cdhash', 'filename|authentihash', 'filename|ssdeep', 'filename|imphash', 'filename|impfuzzy', 'filename|pehash', 'filename|sha224', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|tlsh', 'windows-scheduled-task', 'windows-service-name', 'windows-service-displayname', 'whois-registrant-email', 'whois-registrant-phone', 'whois-registrant-name', 'whois-registrant-org', 'whois-registrar', 'whois-creation-date', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'dns-soa-email', 'size-in-bytes', 'counter', 'datetime', 'cpe', 'port', 'ip-dst|port', 'ip-src|port', 'hostname|port', 'mac-address', 'mac-eui-64', 'email-dst-display-name', 'email-src-display-name', 'email-header', 'email-reply-to', 'email-x-mailer', 'email-mime-boundary', 'email-thread-index', 'email-message-id', 'github-username', 'github-repository', 'github-organisation', 'jabber-id', 'twitter-id', 'first-name', 'middle-name', 'last-name', 'date-of-birth', 'place-of-birth', 'gender', 'passport-number', 'passport-country', 'passport-expiration', 'redress-number', 'nationality', 'visa-number', 'issue-date-of-the-visa', 'primary-residence', 'country-of-residence', 'special-service-request', 'frequent-flyer-number', 'travel-details', 'payment-details', 'place-port-of-original-embarkation', 'place-port-of-clearance', 'place-port-of-onward-foreign-destination', 'passenger-name-record-locator-number', 'mobile-application-id', 'cortex', 'boolean'], 'categories': ['Internal reference', 'Targeting data', 'Antivirus detection', 'Payload delivery', 'Artifacts dropped', 'Payload installation', 'Persistence mechanism', 'Network activity', 'Payload type', 'Attribution', 'External analysis', 'Financial fraud', 'Support Tool', 'Social network', 'Person', 'Other'], 'category_type_mappings': {'Internal reference': ['text', 'link', 'comment', 'other', 'hex'], 'Targeting data': ['target-user', 'target-email', 'target-machine', 'target-org', 'target-location', 'target-external', 'comment'], 'Antivirus detection': ['link', 'comment', 'text', 'hex', 'attachment', 'other'], 'Payload delivery': ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'pehash', 'tlsh', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy', 'filename|pehash', 'mac-address', 'mac-eui-64', 'ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'hostname', 'domain', 'email-src', 'email-dst', 'email-subject', 'email-attachment', 'email-body', 'url', 'user-agent', 'AS', 'pattern-in-file', 'pattern-in-traffic', 'stix2-pattern', 'yara', 'sigma', 'mime-type', 'attachment', 'malware-sample', 'link', 'malware-type', 'comment', 'text', 'hex', 'vulnerability', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'other', 'hostname|port', 'email-dst-display-name', 'email-src-display-name', 'email-header', 'email-reply-to', 'email-x-mailer', 'email-mime-boundary', 'email-thread-index', 'email-message-id', 'mobile-application-id', 'whois-registrant-email'], 'Artifacts dropped': ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy', 'filename|pehash', 'regkey', 'regkey|value', 'pattern-in-file', 'pattern-in-memory', 'pdb', 'stix2-pattern', 'yara', 'sigma', 'attachment', 'malware-sample', 'named pipe', 'mutex', 'windows-scheduled-task', 'windows-service-name', 'windows-service-displayname', 'comment', 'text', 'hex', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'other', 'cookie', 'gene', 'mime-type'], 'Payload installation': ['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'imphash', 'impfuzzy', 'authentihash', 'pehash', 'tlsh', 'cdhash', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha224', 'filename|sha256', 'filename|sha384', 'filename|sha512', 'filename|sha512/224', 'filename|sha512/256', 'filename|authentihash', 'filename|ssdeep', 'filename|tlsh', 'filename|imphash', 'filename|impfuzzy', 'filename|pehash', 'pattern-in-file', 'pattern-in-traffic', 'pattern-in-memory', 'stix2-pattern', 'yara', 'sigma', 'vulnerability', 'attachment', 'malware-sample', 'malware-type', 'comment', 'text', 'hex', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'mobile-application-id', 'other', 'mime-type'], 'Persistence mechanism': ['filename', 'regkey', 'regkey|value', 'comment', 'text', 'other', 'hex'], 'Network activity': ['ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'port', 'hostname', 'domain', 'domain|ip', 'mac-address', 'mac-eui-64', 'email-dst', 'url', 'uri', 'user-agent', 'http-method', 'AS', 'snort', 'pattern-in-file', 'stix2-pattern', 'pattern-in-traffic', 'attachment', 'comment', 'text', 'x509-fingerprint-md5', 'x509-fingerprint-sha1', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'other', 'hex', 'cookie', 'hostname|port', 'bro'], 'Payload type': ['comment', 'text', 'other'], 'Attribution': ['threat-actor', 'campaign-name', 'campaign-id', 'whois-registrant-phone', 'whois-registrant-email', 'whois-registrant-name', 'whois-registrant-org', 'whois-registrar', 'whois-creation-date', 'comment', 'text', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'other', 'dns-soa-email'], 'External analysis': ['md5', 'sha1', 'sha256', 'filename', 'filename|md5', 'filename|sha1', 'filename|sha256', 'ip-src', 'ip-dst', 'ip-dst|port', 'ip-src|port', 'mac-address', 'mac-eui-64', 'hostname', 'domain', 'domain|ip', 'url', 'user-agent', 'regkey', 'regkey|value', 'AS', 'snort', 'bro', 'pattern-in-file', 'pattern-in-traffic', 'pattern-in-memory', 'vulnerability', 'attachment', 'malware-sample', 'link', 'comment', 'text', 'x509-fingerprint-sha1', 'x509-fingerprint-md5', 'x509-fingerprint-sha256', 'ja3-fingerprint-md5', 'github-repository', 'other', 'cortex'], 'Financial fraud': ['btc', 'xmr', 'iban', 'bic', 'bank-account-nr', 'aba-rtn', 'bin', 'cc-number', 'prtn', 'phone-number', 'comment', 'text', 'other', 'hex'], 'Support Tool': ['link', 'text', 'attachment', 'comment', 'other', 'hex'], 'Social network': ['github-username', 'github-repository', 'github-organisation', 'jabber-id', 'twitter-id', 'email-src', 'email-dst', 'comment', 'text', 'other', 'whois-registrant-email'], 'Person': ['first-name', 'middle-name', 'last-name', 'date-of-birth', 'place-of-birth', 'gender', 'passport-number', 'passport-country', 'passport-expiration', 'redress-number', 'nationality', 'visa-number', 'issue-date-of-the-visa', 'primary-residence', 'country-of-residence', 'special-service-request', 'frequent-flyer-number', 'travel-details', 'payment-details', 'place-port-of-original-embarkation', 'place-port-of-clearance', 'place-port-of-onward-foreign-destination', 'passenger-name-record-locator-number', 'comment', 'text', 'other', 'phone-number', 'identity-card-number'], 'Other': ['comment', 'text', 'other', 'size-in-bytes', 'counter', 'datetime', 'cpe', 'port', 'float', 'hex', 'phone-number', 'boolean']}}}
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://<url>/events/5d89ea9a-0778-4a90-a4d0-05a7ac1002c7
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.7', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'm3uBBfGH94cDSSsDvZv7Doqpihgsjg1Zc3WPYs5r', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'Event': {'id': '9', 'orgc_id': '1', 'org_id': '1', 'date': '2019-09-24', 'threat_level_id': '3', 'info': "Today's event", 'published': False, 'uuid': '5d89ea9a-0778-4a90-a4d0-05a7ac1002c7', 'attribute_count': '6', 'analysis': '0', 'timestamp': '1569402304', 'distribution': '1', 'proposal_email_lock': False, 'locked': False, 'publish_timestamp': '0', 'sharing_group_id': '0', 'disable_correlation': False, 'extends_uuid': '', 'event_creator_email': 'admin@admin.test', 'Org': {'id': '1', 'name': '--', 'uuid': '5c34585c-ac5c-46bd-bafa-45570c8b271e'}, 'Orgc': {'id': '1', 'name': '--', 'uuid': '5c34585c-ac5c-46bd-bafa-45570c8b271e'}, 'Attribute': [], 'ShadowAttribute': [], 'RelatedEvent': [], 'Galaxy': [], 'Object': [{'id': '10', 'name': 'file', 'meta-category': 'file', 'description': 'File object describing a file with meta-information', 'template_uuid': '688c46fb-5edb-40a3-8273-1af7923e2215', 'template_version': '15', 'event_id': '9', 'uuid': '5d89f086-8630-4aed-b010-03d9ac1002c7', 'timestamp': '1569321094', 'distribution': '5', 'sharing_group_id': '0', 'comment': 'dsf', 'deleted': False, 'ObjectReference': [], 'Attribute': [{'id': '52', 'type': 'malware-sample', 'category': 'Payload delivery', 'to_ids': True, 'uuid': '5d89f086-defc-4241-a463-03d9ac1002c7', 'event_id': '9', 'distribution': '5', 'timestamp': '1569321094', 'comment': '', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': False, 'object_id': '10', 'object_relation': 'malware-sample', 'value': 'MISP Tags.png|ec90a596bdb7cf99857427ec0934cded', 'Galaxy': [], 'data': '<data>', 'ShadowAttribute': [], 'Tag': [{'id': '1', 'name': 'MOPR_Benign', 'colour': '#00ff19', 'exportable': True, 'user_id': '0', 'hide_tag': False, 'numerical_value': None}]}, {'id': '53', 'type': 'filename', 'category': 'Payload delivery', 'to_ids': False, 'uuid': '5d89f086-88a4-455f-a154-03d9ac1002c7', 'event_id': '9', 'distribution': '5', 'timestamp': '1569321094', 'comment': '', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': False, 'object_id': '10', 'object_relation': 'filename', 'value': 'MISP Tags.png', 'Galaxy': [], 'ShadowAttribute': []}, {'id': '54', 'type': 'md5', 'category': 'Payload delivery', 'to_ids': True, 'uuid': '5d89f086-6aa8-4638-9b6a-03d9ac1002c7', 'event_id': '9', 'distribution': '5', 'timestamp': '1569321094', 'comment': '', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': False, 'object_id': '10', 'object_relation': 'md5', 'value': 'ec90a596bdb7cf99857427ec0934cded', 'Galaxy': [], 'ShadowAttribute': []}, {'id': '55', 'type': 'sha1', 'category': 'Payload delivery', 'to_ids': True, 'uuid': '5d89f086-64f0-4a56-80b3-03d9ac1002c7', 'event_id': '9', 'distribution': '5', 'timestamp': '1569321094', 'comment': '', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': False, 'object_id': '10', 'object_relation': 'sha1', 'value': '1f170b71d5c6f0d5c36593a890c4717bff06cd2e', 'Galaxy': [], 'ShadowAttribute': []}, {'id': '56', 'type': 'sha256', 'category': 'Payload delivery', 'to_ids': True, 'uuid': '5d89f086-7450-455f-b2eb-03d9ac1002c7', 'event_id': '9', 'distribution': '5', 'timestamp': '1569321094', 'comment': '', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': False, 'object_id': '10', 'object_relation': 'sha256', 'value': '7d25eeb7bfa3c66ee2123c5c5cd99c141318e44bae0022bc824467b71cb52a00', 'Galaxy': [], 'ShadowAttribute': []}, {'id': '57', 'type': 'size-in-bytes', 'category': 'Other', 'to_ids': False, 'uuid': '5d89f086-efd8-4594-a197-03d9ac1002c7', 'event_id': '9', 'distribution': '5', 'timestamp': '1569321094', 'comment': '', 'sharing_group_id': '0', 'deleted': False, 'disable_correlation': True, 'object_id': '10', 'object_relation': 'size-in-bytes', 'value': '3159', 'Galaxy': [], 'ShadowAttribute': []}]}], 'Tag': [{'id': '4', 'name': 'Signal', 'colour': '#0057ff', 'exportable': True, 'user_id': '0', 'hide_tag': False, 'numerical_value': None}, {'id': '1', 'name': 'MOPR_Benign', 'colour': '#00ff19', 'exportable': True, 'user_id': '0', 'hide_tag': False, 'numerical_value': None}]}}
Adding MOPR-report Object
DEBUG [aping.py:2073 - _prepare_request() ] POST - http://<url>/objects/add/5d89ea9a-0778-4a90-a4d0-05a7ac1002c7
DEBUG [aping.py:2075 - _prepare_request() ] <MISPObject(name=mopr-report)
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.7', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Length': '833', 'Authorization': 'm3uBBfGH94cDSSsDvZv7Doqpihgsjg1Zc3WPYs5r', 'content-type': 'application/json'}
ERROR [aping.py:2039 - _check_response() ] Something went wrong (403): {'name': 'Could not add object', 'message': 'Could not add object', 'url': '/objects/add/9/', 'errors': 'No valid template found to edit the object.'}
kovacsbalu commented 5 years ago

My debug output is only differ in this request: Your: GET - http://<url>/events/5d89ea9a-0778-4a90-a4d0-05a7ac1002c7 My: POST - https://misp.url/events/restSearch and the result :)

thomasb454 commented 5 years ago

@kovacsbalu That's strange, could you try on the same version as me and report the results?

kovacsbalu commented 5 years ago

@thomasb454 I created MISP 2.4.99 in docker env and I could reproduce

DEBUG [aping.py:2073 - _prepare_request() ] GET - http://localhost/servers/getPyMISPVersion.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'gY0n252vbcOi8EHgmM0xmw7eEGGhEwI9TfUmUoS7', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'version': '2.4.98'}
INFO [aping.py:79 - __init__() ] The version of PyMISP recommended by the MISP instance (response['version']) is older than the one you're using now (2.4.114). If you have a problem, please upgrade the MISP instance or use an older PyMISP version.
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://localhost/servers/getVersion.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'gY0n252vbcOi8EHgmM0xmw7eEGGhEwI9TfUmUoS7', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'version': '2.4.99', 'perm_sync': True}
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://localhost/attributes/describeTypes.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
....
DEBUG [aping.py:2073 - _prepare_request() ] POST - http://localhost/objects/add/5d8d10bc-1b9c-4379-b46e-0034ac130003
DEBUG [aping.py:2075 - _prepare_request() ] <MISPObject(name=myobj)
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Length': '456', 'Authorization': 'gY0n252vbcOi8EHgmM0xmw7eEGGhEwI9TfUmUoS7', 'content-type': 'application/json'}
ERROR [aping.py:2039 - _check_response() ] Something went wrong (403): {'name': 'Could not add object', 'message': 'Could not add object', 'url': '/objects/add/1/', 'errors': 'No valid template found to edit the object.'}

So probably you need to upgrade your MISP. I will try the same with the latest version 2.4.116

kovacsbalu commented 5 years ago

MISP 2.4.116

DEBUG [aping.py:2073 - _prepare_request() ] GET - http://localhost/servers/getPyMISPVersion.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'ISSB8xUulacfME7NyUUSPyT9eWjr9rW3WSTLrWpJ', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'version': '2.4.114'}
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://localhost/servers/getVersion.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'ISSB8xUulacfME7NyUUSPyT9eWjr9rW3WSTLrWpJ', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'version': '2.4.116', 'perm_sync': True}
DEBUG [aping.py:2073 - _prepare_request() ] GET - http://localhost/attributes/describeTypes.json
DEBUG [aping.py:2075 - _prepare_request() ] {}
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Authorization': 'ISSB8xUulacfME7NyUUSPyT9eWjr9rW3WSTLrWpJ', 'content-type': 'application/json'}
.....
DEBUG [aping.py:2073 - _prepare_request() ] POST - http://localhost/objects/add/5d8d1511-c514-4d05-b6b2-010aac180003
DEBUG [aping.py:2075 - _prepare_request() ] <MISPObject(name=myobj)
DEBUG [aping.py:2100 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.114 - Python 3.6', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Length': '456', 'Authorization': 'ISSB8xUulacfME7NyUUSPyT9eWjr9rW3WSTLrWpJ', 'content-type': 'application/json'}
DEBUG [aping.py:2047 - _check_response() ] {'Object': {'id': '1', 'name': 'myobj', 'meta-category': 'misc', 'description': 'myobj details', 'template_uuid': 'a41d6985-ec35-4930-9565-a4fef440b616', 'template_version': '1', 'event_id': '1', 'uuid': '3b295fb3-42cc-4496-bfe1-e1537e59d692', 'timestamp': '1569527129', 'distribution': '5', 'sharing_group_id': '0', 'comment': '', 'deleted': False, 'Attribute': [{'id': '1', 'event_id': '1', 'object_id': '1', 'object_relation': 'url', 'category': 'Network activity', 'type': 'url', 'value1': 'valami.hu', 'value2': '', 'to_ids': True, 'uuid': '67434fb0-dab0-4966-8614-2dc88f261528', 'timestamp': '1569527129', 'distribution': '5', 'sharing_group_id': '0', 'comment': '', 'deleted': False, 'disable_correlation': False, 'value': 'valami.hu'}]}}
thomasb454 commented 5 years ago

Hi @kovacsbalu, Based on the output from MISP 2.4.116 it was successful?

kovacsbalu commented 5 years ago

Yes, and as I wrote before also works with 2.4.109

thomasb454 commented 5 years ago

Ahh I see. I'm not in a position to update my MISP instance currently - for now I'll use the workaround I mentioned and when my MISP instance is upgraded I'll try again. Thank you for the support.

From: Kovács Bálint notifications@github.com Sent: Friday, September 27, 2019 11:26:20 AM To: MISP/PyMISP PyMISP@noreply.github.com Cc: Thomas Bisof thomas2556@live.co.uk; Mention mention@noreply.github.com Subject: Re: [MISP/PyMISP] Adding custom object to event (#462)

Yes, and as I wrote before also works with 2.4.109

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMISP%2FPyMISP%2Fissues%2F462%3Femail_source%3Dnotifications%26email_token%3DAAYT65ECLQGDLHFB5X75PYLQLXNUZA5CNFSM4I2K3PRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7YPEII%23issuecomment-535884321&data=02%7C01%7C%7C6e07652dc93742eadfab08d743352426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637051767826347582&sdata=AMCsFTPw5AMbi9akQARKvy2h%2FT0%2FZH5Rb9qtMKvjnxI%3D&reserved=0, or mute the threadhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAYT65ECU3Q23MY36IHRUY3QLXNUZANCNFSM4I2K3PRA&data=02%7C01%7C%7C6e07652dc93742eadfab08d743352426%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637051767826357593&sdata=0pwWJJFm3h2CjMmiKvgjcXDPhDWdCDsdIPrQBf2P%2B3k%3D&reserved=0.

thomasb454 commented 5 years ago

Hi,

After revisiting this I believe I found the cause of this bug. PyMISP version: 2.4.117.2

My code is as follows:

for result in results:
    new_event = MISPEvent()
    new_event.extends_uuid = result.uuid
    new_event.info = "TESTING EXTENDS"
    res = misp.add_event(new_event)
    new_uuid = res['Event']['uuid']
    new_id = res['Event']['id']

    mopr_obj = MISPObject(name='mopr-report', misp_objects_path_custom='mopr')
    mopr_obj.add_attribute('score', value=result.score)

    misp.add_object(new_id, misp_object=mopr_obj)

This doesn't work and produces the following debug output:

DEBUG [aping.py:2192 - _prepare_request() ] POST - http://<host>/objects/add/33
DEBUG [aping.py:2194 - _prepare_request() ] <MISPObject(name=mopr-report)
DEBUG [aping.py:2219 - _prepare_request() ] {'User-Agent': 'PyMISP 2.4.117.2 - Python 3.7', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Length': '664', 'Authorization': '<key>', 'content-type': 'application/json'}
ERROR [aping.py:2158 - _check_response() ] Something went wrong (403): {'name': 'Could not add object', 'message': 'Could not add object', 'url': '/objects/add/33/', 'errors': 'No valid template found to edit the object.'}

As you can see it's sending the request to /objects/add/33 - but if you try to do the same via the MISP UI it will send the request to /objects/add/[event id]/[object template id].

I fixed this issue by changing the following line in aping.py (line 280) BEFORE:

        new_object = self._prepare_request('POST', f'objects/add/{event_id}', data=misp_object)

AFTER:

        new_object = self._prepare_request('POST', f'objects/add/{event_id}/116', data=misp_object)

In this case I hard-coded 116, which is the ID of my object template - when fixed this obviously needs to be dynamically applied.

Rafiot commented 4 years ago

You should never pass the template ID when you're using PyMISP, I'm not sure how you end-up in this situation.

I patched ExpandedPyMISP to print the json blob and make the debug easier, but canyou show me the content of mopr_obj?

Rafiot commented 4 years ago

This commit may also solve your problem: https://github.com/MISP/MISP/commit/e4c82eb9ff440220be27130bcbcf2de2102e7e35

thomasb454 commented 4 years ago

Hi, My original work around was not for ExpandedPyMISP - just PyMISP. I'm out of the office right now so I'll have to get back to you Monday.

Rafiot commented 4 years ago

Oh, right, PyMISP will not be fixed at this point, as it will go away in ~45 days.