search

MaltegoTech / maltego-trx

Maltego Transform library for Python
MIT License
225 stars 47 forks source link

.setNote() and .addProperty() crash any transform they are used in #6

Closed Nocommas555 closed 4 years ago

Nocommas555 commented 4 years ago

When using methods like .setNote() and .addProperty() the transform crashes saying

ERROR:maltego.server:bad character range This error comes from remove_invalid_xml_chars in python, even when I try to set the Note to "", leading me to believe that the code responsible for converting entities to xml is buggy. The error comes from this line (removing .setNote fixes it)

response.addEntity("maltego.Company", "name").setNote("")

ghost commented 4 years ago

That bad character range error pretty much suggests a regex error. Are you trying to populate that field with a regular expression?

Nocommas555 commented 4 years ago

Some details: 1) I was running it with python2. The same code on python3 worked. 2) If i put a try/except block around this code, the error still gets triggered and the except doesn't get triggered. 3) In the traceback you can see that the error is triggered inside of /Library/Python/2.7/site-packages/maltego_trx/utils.py , which itself is called from the xml-building code 4) If I remove the .setNote, the error is gone, yet an empty string does not have xml banned chars

The error with the traceback: ERROR:maltego.server:bad character range Traceback (most recent call last): File "/Library/Python/2.7/site-packages/maltego_trx/server.py", line 45, in run_transform return transform_method.run_transform(client_msg), 200 # Transform class File "/Library/Python/2.7/site-packages/maltego_trx/transform.py", line 13, in run_transform return response.returnOutput() File "/Library/Python/2.7/site-packages/maltego_trx/maltego.py", line 184, in returnOutput lines.append(entity.returnEntity()) File "/Library/Python/2.7/site-packages/maltego_trx/maltego.py", line 136, in returnEntity lines.append(self.add_field_to_xml(additional_field)) File "/Library/Python/2.7/site-packages/maltego_trx/maltego.py", line 111, in add_field_to_xml "value": remove_invalid_xml_chars(value), File "/Library/Python/2.7/site-packages/maltego_trx/utils.py", line 47, in remove_invalid_xml_chars val = re.sub(u'[^\u0020-\uD7FF\u0009\u000A\u000D\uE000-\uFFFD\U00010000-\U0010FFFF]+', '?', val) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 155, in sub return _compile(pattern, flags).sub(repl, string, count) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py", line 251, in _compile raise error, v # invalid expression

jb731 commented 4 years ago

FWIW I've experienced this as well. As far as I can tell, the issue is specifically \U00010000-\U0010FFFF in the re.sub function. I'm not sure what characters those are, but apparently python2 does not like them, but python3 seems to tolerate them.

ghost commented 4 years ago

Definitely a problem with Unicode, regular expressions, and his support in python2.

Is hard to see the exact point of the problem without looking at the code, but as a rule of thumb, try to convert the string to unicode string before pushing the string to the maltego object.

    u = unichr(40960) + u'abcd' + unichr(1972)
    utf8_version = u.encode('utf-8')
    response.addEntity("maltego.Company", utf8_version).setNote("")

alternatively, try to use iso-8859-1 encoding instead, since it prompted problems to other users as well.

Bottom line: Python2 is coming to an end, so better think dropping py2 compatibility in your projects :) https://docs.python.org/2/howto/unicode.html