Upgrade to python 3 and Angr 9.0

[cpbscholten]

This pull request updates the Karonte framework to Python 3 and Angr 9.0 with many other fixes and changes. I have used a modified version of Karonte for my master's thesis and using this PR I want to contribute back my improvements to the framework.

It contains a lot of changes, but the changes are so intertwined that it became impossible to split up in multiple PR's.

Changes

• Upgrade to Angr 9.0
• Upgrade to Python 3
• Replaced the extractor with the Firmadyne extractor
• The buffer sizes are now analyzed using the bit vectors. This allows us to discover the size of stack and heap variables
• The algorithm for indirect string references was deprecated and now uses the XREFManager of Angr
• The taint propagation has been improved
• Added new rules for when to raise alerts in sinks.py
• Speed and memory improvements (e.g. only calculate each CFG once in the BBF)
• All dependencies have been documented in a requirements.txt file
• Removed duplicate code

There are still some small issues remaining:

• First, the provided logger in the code was broken, so this version cannot log .json, but only the .txt based version
• The algorithm to pick the highest scoring cluster of binaries in the BBF is broken. I have something working, but this requires further review. (It now usually picks too many binaries)

[JonathanGoikhman]

Hi, thanks for your work. I do have a problem when running karonte.py. I get the error: self._pickle_parsers = self._config['pickle_parsers'] KeyError: 'pickle_parsers' This field can be empty, isn't he? When I run the tool, i get: INFO | 2021-08-21 08:35 | Karonte | Extracting firmware image. This may take a while... INFO | 2021-08-21 08:35 | Karonte | Logging at: /tmp/Karonte.txt_42 INFO | 2021-08-21 08:35 | Karonte | Firmware directory: /tmp/fw/DIR823A1_FW100WWb05.bin INFO | 2021-08-21 08:35 | Karonte | Retrieving Border Binaries ERROR | 2021-08-21 08:35 | Karonte | No border binaries found, exiting... INFO | 2021-08-21 08:35 | Karonte | Finished, results in /tmp/Karonte.txt_42 0

[zardus]

@badnack @Machiry , any objections to merging this in? I guess the current version no longer works anyways. If we merge this in and add testcases, we could keep karonte alive forever :-)

[cpbscholten]

Hi, thanks for your work. I do have a problem when running karonte.py. I get the error: self._pickle_parsers = self._config['pickle_parsers'] KeyError: 'pickle_parsers' This field can be empty, isn't he? When I run the tool, i get: INFO | 2021-08-21 08:35 | Karonte | Extracting firmware image. This may take a while... INFO | 2021-08-21 08:35 | Karonte | Logging at: /tmp/Karonte.txt_42 INFO | 2021-08-21 08:35 | Karonte | Firmware directory: /tmp/fw/DIR823A1_FW100WWb05.bin INFO | 2021-08-21 08:35 | Karonte | Retrieving Border Binaries ERROR | 2021-08-21 08:35 | Karonte | No border binaries found, exiting... INFO | 2021-08-21 08:35 | Karonte | Finished, results in /tmp/Karonte.txt_42 0

Right now, just as in the current version of Karonte, the pickle_parsers cannot be empty.

Did you install the dependencies for the extractor? See: https://github.com/cpbscholten/karonte/blob/master/tool/libraries/extractor/README.md

[JonathanGoikhman]

Hi, thanks for your work. I do have a problem when running karonte.py. I get the error: self._pickle_parsers = self._config['pickle_parsers'] KeyError: 'pickle_parsers' This field can be empty, isn't he? When I run the tool, i get: INFO | 2021-08-21 08:35 | Karonte | Extracting firmware image. This may take a while... INFO | 2021-08-21 08:35 | Karonte | Logging at: /tmp/Karonte.txt_42 INFO | 2021-08-21 08:35 | Karonte | Firmware directory: /tmp/fw/DIR823A1_FW100WWb05.bin INFO | 2021-08-21 08:35 | Karonte | Retrieving Border Binaries ERROR | 2021-08-21 08:35 | Karonte | No border binaries found, exiting... INFO | 2021-08-21 08:35 | Karonte | Finished, results in /tmp/Karonte.txt_42 0

Right now, just as in the current version of Karonte, the pickle_parsers cannot be empty.

Did you install the dependencies for the extractor? See: https://github.com/cpbscholten/karonte/blob/master/tool/libraries/extractor/README.md

Yes, I can run this script no problems. So how should I run the tool (What do I specify in pickle_parsers field)? If I put the blank space this is the output, If i leave it empy I get 'KeyError' because it's empty. Is there a way to get the pk file?

[conand]

Merged. Thanks, Christian @cpbscholten!