bworrell
commented
10 years ago I took a pass at adding parameters to to_xml() to allow developers to bypass the automatic collection of namespaces in hopes that it would improve performance a bit.
The https://github.com/STIXProject/python-stix/commit/820931838676b0fb027b6fc4a5860db3e68807e2 commit introduces an auto_namespace parameter with a default value of True. When set to False, the namespaces and schemalocations are all pulled from the ns_dict and schemaloc_dict parameters.
I ran the following script to get a feel for the performance difference:
#!/usr/bin/env python
from stix.core import STIXPackage
FN = "Appendix_G_IOCs_Full.xml"
print "[-] Parsing", FN
PACKAGE = STIXPackage.from_xml(FN)
NS_DICT = {
'http://cybox.mitre.org/objects#WinDriverObject-3': 'WinDriverObj',
'http://stix.mitre.org/common-1': 'stixCommon',
'http://data-marking.mitre.org/extensions/MarkingStructure#Terms_Of_Use-1': 'TOUMarking',
'http://www.w3.org/2001/XMLSchema-instance': 'xsi',
'http://cybox.mitre.org/objects#WinProcessObject-2': 'WinProcessObj',
'http://cybox.mitre.org/common-2': 'cyboxCommon',
'http://cybox.mitre.org/objects#WinServiceObject-2': 'WinServiceObj',
'http://cybox.mitre.org/objects#FileObject-2': 'FileObj',
'http://cybox.mitre.org/objects#ProcessObject-2': 'ProcessObj',
'http://stix.mitre.org/stix-1': 'stix',
'http://cybox.mitre.org/cybox-2': 'cybox',
'http://stix.mitre.org/Incident-1': 'incident',
'http://data-marking.mitre.org/Marking-1': 'marking',
'http://cybox.mitre.org/default_vocabularies-2': 'cyboxVocabs',
'http://stix.mitre.org/extensions/TestMechanism#OpenIOC2010-1': 'stix-openioc',
'http://stix.mitre.org/default_vocabularies-1': 'stixVocabs',
'http://cybox.mitre.org/objects#WinRegistryKeyObject-2': 'WinRegistryKeyObj',
'http://cybox.mitre.org/objects#WinExecutableFileObject-2': 'WinExecutableFileObj',
'http://example.com': 'example',
'http://cybox.mitre.org/objects#WinFileObject-2': 'WinFileObj',
'http://cybox.mitre.org/objects#WinHandleObject-2': 'WinHandleObj',
'http://www.mandiant.com': 'mandiant',
'http://stix.mitre.org/TTP-1': 'ttp',
'http://stix.mitre.org/Indicator-2': 'indicator'
}
SCHEMALOC_DICT = {
'http://cybox.mitre.org/common-2': 'http://cybox.mitre.org/XMLSchema/common/2.1/cybox_common.xsd',
'http://cybox.mitre.org/cybox-2': 'http://cybox.mitre.org/XMLSchema/core/2.1/cybox_core.xsd',
'http://cybox.mitre.org/default_vocabularies-2': 'http://cybox.mitre.org/XMLSchema/default_vocabularies/2.1/cybox_default_vocabularies.xsd',
'http://cybox.mitre.org/objects#FileObject-2': 'http://cybox.mitre.org/XMLSchema/objects/File/2.1/File_Object.xsd',
'http://cybox.mitre.org/objects#ProcessObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Process/2.1/Process_Object.xsd',
'http://cybox.mitre.org/objects#WinDriverObject-2': '../../cybox/objects/Win_Driver_Object.xsd',
'http://cybox.mitre.org/objects#WinDriverObject-3': 'http://cybox.mitre.org/XMLSchema/objects/Win_Driver/3.0/Win_Driver_Object.xsd',
'http://cybox.mitre.org/objects#WinExecutableFileObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Win_Executable_File/2.1/Win_Executable_File_Object.xsd',
'http://cybox.mitre.org/objects#WinFileObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Win_File/2.1/Win_File_Object.xsd',
'http://cybox.mitre.org/objects#WinHandleObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Win_Handle/2.1/Win_Handle_Object.xsd',
'http://cybox.mitre.org/objects#WinProcessObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Win_Process/2.1/Win_Process_Object.xsd',
'http://cybox.mitre.org/objects#WinRegistryKeyObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Win_Registry_Key/2.1/Win_Registry_Key_Object.xsd',
'http://cybox.mitre.org/objects#WinServiceObject-2': 'http://cybox.mitre.org/XMLSchema/objects/Win_Service/2.1/Win_Service_Object.xsd',
'http://data-marking.mitre.org/Marking-1': 'http://stix.mitre.org/XMLSchema/data_marking/1.1.1/data_marking.xsd',
'http://data-marking.mitre.org/extensions/MarkingStructure#Terms_Of_Use-1': 'http://stix.mitre.org/XMLSchema/extensions/marking/terms_of_use/1.0/terms_of_use_marking.xsd',
'http://stix.mitre.org/Incident-1': 'http://stix.mitre.org/XMLSchema/incident/1.1.1/incident.xsd',
'http://stix.mitre.org/Indicator-2': 'http://stix.mitre.org/XMLSchema/indicator/2.1.1/indicator.xsd',
'http://stix.mitre.org/TTP-1': 'http://stix.mitre.org/XMLSchema/ttp/1.1.1/ttp.xsd',
'http://stix.mitre.org/common-1': 'http://stix.mitre.org/XMLSchema/common/1.1.1/stix_common.xsd',
'http://stix.mitre.org/default_vocabularies-1': 'http://stix.mitre.org/XMLSchema/default_vocabularies/1.1.1/stix_default_vocabularies.xsd',
'http://stix.mitre.org/extensions/TestMechanism#OpenIOC2010-1': 'http://stix.mitre.org/XMLSchema/extensions/test_mechanism/open_ioc_2010/1.1.1/open_ioc_2010_test_mechanism.xsd',
'http://stix.mitre.org/extensions/Test_Mechanism#OpenIOC2010-1': '../../extensions/test_mechanism/open_ioc_2010_test_mechanism.xsd',
'http://stix.mitre.org/stix-1': 'http://stix.mitre.org/XMLSchema/core/1.1.1/stix_core.xsd'
}
def test_auto_namespace():
foo = PACKAGE.to_xml()
def test_explicit_namespace():
foo = PACKAGE.to_xml(auto_namespace=False, ns_dict=NS_DICT,
schemaloc_dict=SCHEMALOC_DICT)
if __name__ == "__main__":
from timeit import Timer
t_auto = Timer("test_auto_namespace()", "from __main__ import test_auto_namespace")
print "test_auto_namespace: ", t_auto.repeat(5, 1)
t_explicit = Timer("test_explicit_namespace()", "from __main__ import test_explicit_namespace")
print "test_explicit_namespace: ", t_explicit.repeat(5, 1)Which resulted with:
$ python -W ignore ./time_turnaround.py
[-] Parsing Appendix_G_IOCs_Full.xml
test_auto_namespace: [3.804380178451538, 3.7833778858184814, 3.786379098892212, 3.7823779582977295, 3.7763779163360596]
test_explicit_namespace: [1.4481451511383057, 1.4481439590454102, 1.4461450576782227, 1.4471449851989746, 1.4501450061798096]The test_auto_namespace() method used all default to_xml() params and usually took about 3.8 seconds to run. The test_explicit_namespace() method used pre-crafted namespace and schemalocation dictionaries and passed them into to_xml(), also setting auto_namespace=False which averaged around 1.45 seconds to run.
I used the Mandiant APT1 Full sample document (also used in the cProfile run) for these tests. It seems like a big improvement if you know exactly what namespaces you're going to be exporting.
Note: I've never used timeit before..so I could have totally misused it :)
gtback
iglocska
usrlocalben
The big bottleneck of to_xml() is the recursive inspection for namespaces: