Attempting to use shellcode hash search for 64-bit IDBs may elicit the following error (hand-typed, HOPEFULLY free of transcription errors):
Traceback (most recent call last):
File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 297, in lookForOpArgs
hits = self.dbstore.getSymbolByTypeHash(h.hashType, opval)
File "C:/Program Files/IDA 7.1/python/flare/shellcode_hash_search.py", line 184, in getSymbolByTypeHash
cur = self.conn.execute(sql_lookup_hash_type_value, (hashVal, hashType))
OverflowError: Python int too large to convert to SQLite INTEGER
Commonly, 64-bit shellcode uses 32-bit hash algorithms with 32-bit registers, so one workaround might be to check that the upper 32 bits are null or all ones (sign-extended negative quantity) and truncate (respecting sign extension) before querying (or raise an exception if the upper 32 bits contain numeric information of any significance. It might be better to promote the db schema and existing hashes to 64-bit quantities and make this work seamlessly for 32- and 64-bit IDBs if possible.
Attempting to use shellcode hash search for 64-bit IDBs may elicit the following error (hand-typed, HOPEFULLY free of transcription errors):
Commonly, 64-bit shellcode uses 32-bit hash algorithms with 32-bit registers, so one workaround might be to check that the upper 32 bits are null or all ones (sign-extended negative quantity) and truncate (respecting sign extension) before querying (or raise an exception if the upper 32 bits contain numeric information of any significance. It might be better to promote the db schema and existing hashes to 64-bit quantities and make this work seamlessly for 32- and 64-bit IDBs if possible.