Could not load stix file (Probably huge/deep XML tree)

[Danko90]

Hi,

I ran into an error while trying to pull data from a TAXII Server in STIX Format. This is the traceback

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 156, in manage_stix
    stix_package = STIXPackage.from_xml(stix)
  File "/usr/local/lib/python3.4/dist-packages/stix/core/stix_package.py", line 249, in from_xml
    return entity_parser.parse_xml(xml_file, encoding=encoding)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/parser.py", line 179, in parse_xml
    xml_etree = get_etree(xml_file, encoding=encoding)
  File "/usr/local/lib/python3.4/dist-packages/mixbox/xml.py", line 55, in get_etree
    return etree.parse(doc, parser=parser)
  File "src/lxml/lxml.etree.pyx", line 3427, in lxml.etree.parse (src/lxml/lxml.etree.c:81117)
  File "src/lxml/parser.pxi", line 1832, in lxml.etree._parseDocument (src/lxml/lxml.etree.c:118116)
  File "src/lxml/parser.pxi", line 1852, in lxml.etree._parseFilelikeDocument (src/lxml/lxml.etree.c:118399)
  File "src/lxml/parser.pxi", line 1747, in lxml.etree._parseDocFromFilelike (src/lxml/lxml.etree.c:117187)
  File "src/lxml/parser.pxi", line 1162, in lxml.etree._BaseParser._parseDocFromFilelike (src/lxml/lxml.etree.c:111914)
  File "src/lxml/parser.pxi", line 595, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:105109)
  File "src/lxml/parser.pxi", line 706, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:106817)
  File "src/lxml/parser.pxi", line 635, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:105671)
  File "/home/misp/MISP-Taxii-Server/fsisac.xml", line 611
lxml.etree.XMLSyntaxError: AttValue: ' expected, line 611, column 87

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
    return manage_stix(f)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
    raise STIXLoadError("Could not load stix file. {0}".format(ex))
misp_stix_converter.errors.STIXLoadError: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.4/dist-packages/Flask-0.12.1-py3.4.egg/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 76, in wrapper
    return _process_with_service(service)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/middleware.py", line 154, in _process_with_service
    response_message = service.process(request.headers, taxii_message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 89, in process
    in_response_to=message.message_id)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/exceptions.py", line 48, in raise_failure
    tb=tb)
  File "/usr/local/lib/python3.4/dist-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/abstract.py", line 83, in process
    response_message = handler.handle_message(self, message)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 126, in handle_message
    return InboxMessage11Handler.handle_message(service, request)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/taxii/services/handlers/inbox_message_handlers.py", line 65, in handle_message
    inbox_message_id=inbox_message.id if inbox_message else None)
  File "/usr/local/lib/python3.4/dist-packages/opentaxii-0.1.9a1-py3.4.egg/opentaxii/persistence/manager.py", line 164, in create_content
    collection_ids=collection_ids, service_id=service_id)
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in send
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/blinker-1.4-py3.4.egg/blinker/base.py", line 267, in <listcomp>
    for receiver in self.receivers_for(sender)]
  File "/usr/local/lib/python3.4/dist-packages/misp_taxii_hooks-0.2-py3.4.egg/misp_taxii_hooks/hooks.py", line 59, in post_stix
    package = pymisp.tools.stix.load_stix(content_block.content)
  File "/usr/local/lib/python3.4/dist-packages/pymisp/tools/stix.py", line 17, in load_stix
    stix = convert.load_stix(stix)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 132, in load_stix
    return manage_stix(f)
  File "/usr/local/lib/python3.4/dist-packages/misp_stix_converter-0.2.9-py3.4.egg/misp_stix_converter/converters/convert.py", line 160, in manage_stix
    raise STIXLoadError("Could not load stix file. {0}".format(ex))
opentaxii.taxii.exceptions.FailureStatus: Could not load stix file. AttValue: ' expected, line 611, column 87 (fsisac.xml, line 611), event=Status exception}

I checked the XML and it is correct (unfortunately I can't share it) This is the configuration file

---
domain: "localhost:9000"
support_basic_auth: yes

save_raw_inbox_messages: yes
xml_parser_supports_huge_tree: no

persistence_api:
  class: opentaxii.persistence.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: mysql://taxii:Password1234@localhost/taxiipersist
    create_tables: yes

auth_api:
  class: opentaxii.auth.sqldb.SQLDatabaseAPI
  parameters:
    db_connection: mysql://taxii:Password1234@localhost/taxiiauth
    create_tables: yes
    secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

logging:
  opentaxii: debug
  root: debug

hooks: misp_taxii_hooks.hooks
# Sample configuration for misp_taxii_server

zmq:
    host: 192.168.56.50
    port: 50000

misp:
    url: "https://192.168.56.50"
    api: XXXXXXXXXXXXXXXXXXXXXXXXX

I tried to change the xml_parser_supports_huge_tree option in yes, but nothing changed. The XML block size that causes this error is 65535 bytes. As you can notice on the stacktrace I modified a bit the source code by writing each XML Block in a fixed file in order to see where is the error.

Can anyone help me?

Thanks in advance

[FloatingGhost]

File "/home/misp/MISP-Taxii-Server/fsisac.xml", line 611
  lxml.etree.XMLSyntaxError: AttValue: ' expected, line 611, column 87

your XML is invalid.

This is not an issue with the converter, but I'll to help.

Can you post line 611 from fsisac.xml?

[Danko90]

Hi @FloatingGhost, thanks for helping. I can't share all XML because it contains sensitive data, I can say you that is a very large file (800MB). I tried to download it by using another taxiiclient and it is valid (I used xmllint). The strange thing is that MISP-TAXII-Server receives block by block and when it raises the exception that block is incomplete. This is the 611 line

<WinExecutableFileObj:Function_Name condition="Equa

As you can see it is incomplete, and I can't understand why!

[FloatingGhost]

hum, weird

I don't have access to fsisac, so I can't pull the file, and misp-taxii-server just uses OpenTAXII

I can't say I know why it'd cut off half way through >__<

[Danko90]

@FloatingGhost I tried to upload the whole file by using stix-to-misp.py and I got this exception:

mixbox.parser.UnsupportedRootElementError: Document root element ({http://taxii.mitre.org/messages/taxii_xml_binding-1.1}Poll_Response) not one of (['{http://stix.mitre.org/stix-1}STIX_Package'])

I guess it's related to mixbox, but do you know what kind of error is this?

[FloatingGhost]

That implies that... well, the root element of the document isn't \<STIXPackage>

[Danko90]

Hi @FloatingGhost, I'm still having that problem, I tried to load the XML manually and it works. The problem seems to be located in the connection between the TAXII client and the repository. In particular the TAXII client can't receive all block data from the repository, in the same XML point, which is strange. Do you have any idea?

[FloatingGhost]

I can't test, I have no data and absolutely no access to any taxii servers. I can't help.

Closing.

[Danko90]

I noticed a warning saying:

Data truncated for column 'content' at row 1 

So on mysql I found out that variable has a maximum length of 65535, that's the reason of that error.

select COLUMN_NAME, CHARACTER_MAXIMUM_LENGTH 
from information_schema.columns
where table_schema = DATABASE() AND   
      table_name = 'content_blocks' AND        
      COLUMN_NAME = 'content';

+-------------+--------------------------+
| COLUMN_NAME | CHARACTER_MAXIMUM_LENGTH |
+-------------+--------------------------+
| content     |                    65535 |
+-------------+--------------------------+
1 row in set (0.00 sec)

I modified that variable by setting it as LONGTEXT. Now it works fine. Can you change it? Thanks for your help

[FloatingGhost]

No I cannot change that. That is a feature of OpenTAXII.

Go raise it there.

[Danko90]

This is the output

root@misp-ubuntu:/home/misp/MISP-Taxii-Server/config# mysql --version
mysql  Ver 14.14 Distrib 5.5.54, for debian-linux-gnu (x86_64) using readline 6.3
root@misp-ubuntu:/home/misp/MISP-Taxii-Server/config#

[FloatingGhost]

:P you're using mysql <15

Update your mysql to at least ver15.

[Danko90]

Damn! I'm using the Ubuntu MISP iso.. I didn't install anything since it was already prepared! I'll let you know, thanks a lot!