search

MISP / MISP

MISP (core software) - Open Source Threat Intelligence and Sharing Platform
https://www.misp-project.org/
GNU Affero General Public License v3.0
5.33k stars 1.39k forks source link

Cannot create STIXv2 Event with IP4-value and marking definitions #4179

Closed tm3452 closed 5 years ago

tm3452 commented 5 years ago

Work environment

Questions Answers
Type of issue Bug
OS version (server) ubuntu
OS version (client) Ubuntu
PHP version 7.2
MISP version / git hash MISP 2.4.101

Expected behavior

Event in MISP successfully created

Actual behavior

No event created. Error shown: "Could not import STIX document: Issues executing the ingestion script or invalid input. Please check whether the dependencies for STIX are met via the diagnostic tool."

Steps to reproduce the behavior

Save the STIX 2 sample from: https://oasis-open.github.io/cti-documentation/examples/using-marking-definitions In MISP: Event Actions > List Events > Import From... > STIX 2.0. Select "browse..." and choose the saved sample". Click "upload"

Notes

If you change "ipv4-addr" to "ip-dst" it passes.

adulau commented 5 years ago

Confirmed with the sample

chrisr3d commented 5 years ago

With "ip-dst", which is not valid STIX2 type, you actually get the pattern itself as is because we import it before trying to extract the actual type and value, so if it fails (because it can be really difficult to parse patterns in some cases) you at least get the pattern.

This is exactly what happens here, if I try to import a pattern with 'ip-dst' instead of 'ipv4-addr', it is not recognized and you get a warning in the log file (PATH_TO_MISP/app/tmp/logs/exec-errors.log): ('ip-dst',) not parsed at the moment

I still do not get why it does not fail the same way when you use 'ip-dst' and 'ipv4-addr' but 'ipv4-addr' is the one that we try to map from a STIX2 know type to a MISP type which is indeed ip-dst or ip-src.

What we can look at that point is whether you have the good versions of the python libraries installed, which you can find going through Administration -> Server Settings & Maintenance -> Diagnostic, and scrolling down until you find the section "STIX and Cybox libraries"

tm3452 commented 5 years ago

Not sure we really understand what is going on here, but here are the versions we are using:

STIX: 1.2.0.6 CyBox: 2.1.0.18.dev0 mixbox: 1.0.3 maec: 4.1.0.14 PyMISP: >2.4.93 Other versions might work but are not tested / recommended. Current libraries status…OK STIX library version…OK CYBOX library version…OK MIXBOX library version…OK MAEC library version…OK PYMISP library version…OK

chrisr3d commented 5 years ago

Using the original file provided in the example (with ipv4-addr), when it fails, can you tell me what you get in the log file (YOUR_PATH_TO_MISP/app/tmp/logs/exec-errors.log) ?

tm3452 commented 5 years ago

Here is the error:

Unknown STIX 2.0 type: identity Traceback (most recent call last): File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 1421, in main(sys.argv) File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 1416, in main stix_parser.handler() File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 992, in handler self.general_handler() File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 71, in general_handler self.build_from_STIX_with_report() if self.report else self.build_from_STIX_without_report() File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 117, in build_from_STIX_without_report self.parsing_process(_object, object_type) File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 996, in parsing_process self.object_from_refsobject_type File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 1027, in parse_external_indicator self.parse_external_pattern(pattern, indicator_id, marking=indicator.object_marking_refs) File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 1069, in parse_external_pattern self.pattern_mapping[types](pattern, marking, uuid=uuid) File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 1206, in parse_ip_address_pattern self.add_attributes_from_pattern('ip-dst', pattern, marking, uuid) File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 1126, in add_attributes_from_pattern attribute = self.add_tag_in_attribute(attribute, marking) File "/var/www/MISP/app/files/scripts/stix2/stix2misp.py", line 171, in add_tag_in_attribute attribute.add_tag(self.parse_marking(marking['object'])) File "/var/www/MISP/venv/lib/python3.6/site-packages/pymisp/abstract.py", line 246, in __add_tag raise PyMISPInvalidFormat("The tag is in an invalid format (can be either string, MISPTag, or an expanded dict): {}".format(tag)) pymisp.exceptions.PyMISPInvalidFormat: The tag is in an invalid format (can be either string, MISPTag, or an expanded dict): None

chrisr3d commented 5 years ago

Thank you,

I spotted this issue when I started trying to see what was going on with the 'ipv4-addr' pattern type, and it has been fixed with f3ae3a7.

Can you pull the latest version again and see if it is now working for you ?

tm3452 commented 5 years ago

We've pulled the latest version and I can confirm this is now working in v2.4.102. Many thanks for your assistance this can be closed as far as we are concerned.

chrisr3d commented 5 years ago

Sure, no worries Don't hesitate to poke us again whenever you get something not working as expected