Optimization: Improve IOC extraction efficiency using parallel processing

[kienmarkdo]

IOC extractor is currently very inefficient as it is checking EVERY SINGLE MESSAGE against MULTIPLE Regex patterns, each pattern belonging to an IOC.

This should be sped up.

Attempted solutions

• Search optimization (omit IOC extraction on some messages). All messages need to be analyzed.
• Improve string handling: Most messages are already short. No need to do this.
• Optimize regex patterns: Already optimized.
• Caching results: Texts are not identical. Caching would be useless.

So, use parallel processing.

[kienmarkdo]

Closing as parallel processing won't improve the performance at all

https://stackoverflow.com/questions/9984288/could-threading-or-multiprocessing-improve-performance-when-analyzing-a-single-s

https://stackoverflow.com/questions/18263285/python-compiling-regexes-in-parallel