search

mandiant / flare-ida

IDA Pro utilities from FLARE team
Apache License 2.0
2.23k stars 466 forks source link

OverflowError with 64-bit IDBs #76

Closed strictlymike closed 5 years ago

strictlymike commented 5 years ago

Attempting to use shellcode hash search for 64-bit IDBs may elicit the following error (hand-typed, HOPEFULLY free of transcription errors):

Traceback (most recent call last):
  File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 297, in lookForOpArgs
    hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
  File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 184, in getSymbolByTypeHash
    cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER

Commonly, 64-bit shellcode uses 32-bit hash algorithms with 32-bit registers, so one workaround might be to check that the upper 32 bits are null or all ones (sign-extended negative quantity) and truncate (respecting sign extension) before querying (or raise an exception if the upper 32 bits contain numeric information of any significance. It might be better to promote the db schema and existing hashes to 64-bit quantities and make this work seamlessly for 32- and 64-bit IDBs if possible.

jhsmith commented 5 years ago

Should be resolved in #79. Please try this the new version and let us know if you encounter more issues.