search

MISP / MISP-Taxii-Server

An OpenTAXII Configuration for MISP
BSD 3-Clause "New" or "Revised" License
81 stars 30 forks source link

STIX Parser Crashing After Reading 65535 Bytes #58

Open TheDr1ver opened 5 years ago

TheDr1ver commented 5 years ago

I'm not sure if this is some weird issue with my server or Python or if it's something dealing with one of the underlying libraries used by the MISP Taxii Service, but I'm kind of stumped.

I'm attempting to import STIX files obtained from the Anomali TAXII feed into MISP, but every STIX file I tried was crashing. After banging my head against the wall for a bit, I realized that while lxml is parsing the STIX file it crashes while reading the 65535th character of the STIX file.

Poll Anomali to get the latest Phish Tank STIX file taxii-poll --host limo.anomali.com --https --collection Phish_Tank_F107 --discovery /api/v1/taxii/taxii-discovery-service/ --username guest --password guest --begin 2018-11-01T00:00:00Z --end 2018-12-22T00:00:00Z > phishtank.xml

Attempt to push the STIX file to opentaxii-run-dev taxii-push --path http://localhost:9000/services/inbox -f phishtank.xml --dest collection --username MYUSERNAME --password MYPASSWORD

Error Log - line 847, column 228 is the 65535th char in the STIX file

    127.0.0.1 - - [21/Dec/2018 12:59:04] "POST /services/inbox HTTP/1.1" 200 -
'cm9vdDpyb290' 12
Posting STIX...
2018-12-21T18:00:36.436787Z [opentaxii.middleware] warning: Status exception {exception=Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 112, in load_stix
    stix_package = STIXPackage.from_json(data)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/entities.py", line 495, in from_json
    d = json.loads(json_doc)
  File "/usr/lib/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in proce
ss
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers
.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers
.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_c
ontent
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 62, in post_stix
    package = pymisp.tools.stix.load_stix(StringIO(block))
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 16, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 119, in load_stix
    stixXml = etree.fromstring(stix.read())
  File "src/lxml/etree.pyx", line 3213, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1757, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1067, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 710, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 639, in lxml.etree._raiseParseError
  File "<string>", line 847
lxml.etree.XMLSyntaxError: Premature end of data in tag Description line 847, line 847, column 228

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
    in_response_to=message.message_id)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
    tb=tb)
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.10a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 62, in post_stix
    package = pymisp.tools.stix.load_stix(StringIO(block))
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 16, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter/converters/convert.py", line 119, in load_stix
    stixXml = etree.fromstring(stix.read())
  File "src/lxml/etree.pyx", line 3213, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1876, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1757, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1067, in lxml.etree._BaseParser._parseUnicodeDoc
  File "src/lxml/parser.pxi", line 600, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 710, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 639, in lxml.etree._raiseParseError
opentaxii.taxii.exceptions.FailureStatus: Premature end of data in tag Description line 847, line 847, column 228 (<string>, line 847), logger=opentaxii.middleware, timestamp=2018-12-21T18:00:36.436787Z, event=Status exception, level=warning}

I'm wondering if it has something to do w/ the TEXT data object for MySQL maxing out at 65535 characters, but I'm not sure where I could find the type of data object being used to send the STIX docs to the database and possibly change it to MEDIUMTEXT or LONGTEXT.

Thanks in advance for any help you might be able to provide!

TheDr1ver commented 5 years ago

Actually, now that I'm looking at it I'm wondering if this pull request addressed this very issue:

https://github.com/eclecticiq/OpenTAXII/commit/882e2a50a9c838cc1bf4ef89e42aa63f5544b4a1#diff-ce3f7b939e5c540480ac655aef32c513

Any idea if MISP-Taxii-Server can be successfully linked to the latest version of OpenTAXII without breaking? It looks like the one it's linked to is from June of last year, while the pull request addressing this issue was in October of last year.

FloatingGhost commented 5 years ago

Managed to get around to updating to the latest version of OpenTAXII. Installation instructions and dependencies are updated.

Should probably fix the issue