Bug Fix for IOC File Processing Errors

[rc-csmith]

Changes

• Rather than create a function dedicated for IOC file processing, this code uses the existing nested_process_search functions to generate queries. The nested_process_search functions already handle parameter mapping and can record the source/search used to generate the results
• Added support for MD5 to all EDR products
• Add logic if number of provided IOCs is greater than 100, process them in chunks for S1 and CbC to avoid errors.

Known Issues

• Cortex doesn't collect domain information which will cause errors/warnings for domain fields in IOC files/deffiles

Closes #97

[xC0uNt3r7hr34t]

as an after thought, I realized there is not any process checking for the max number of items that can be in a single search. I believe S1 has a limit of 100 items within a search as well as a total character limit. We may need to consider chunking up these ioc files. I would have to research the other EDR as to if they have similar limits.

[rc-csmith]

I hadn't thought about query character limits but that's it makes sense you could have a giant list of IOCs that need processing. I can help research the Carbon Black ones and see if there's any documentation on that. Depending on which EDR is the most limiting, we could put restrict the size of IOC files within the code - just process the first X lines and then log an error/warning so the user knows. Plus document that limitation in the wiki

[xC0uNt3r7hr34t]

We should be able to parse the data in chunks just as we do with the definition files already. simply need to add a check in for data limits. It would likely apply to the definition files as well. If I were to add 100 process names to a single definition, that would likely cause problems.

MDE doesn't seem to have a limit based on dropping a 100+ IPs in their query and it ran fine. S1 is limited on 100 items per search field and a total of 8192 characters for Deep Visibility. PowerQuery has the same character limitation but can support more than 100 items from my testing of using 271 IPs.

[rc-csmith]

I can't find any official documentation for either CbR nor CbC but from testing,

• CbR has no limits on number of terms and ran just fine with 150 - 200 IPs
• CbC maxed out and threw errors when trying to ship 150 IPs in a single query. Limiting the size to 100 seemed to work, though

[xC0uNt3r7hr34t]

I would recommend only implementing changes to CbC and S1 then since the other EDRs can handle the logic fine.

[rc-csmith]

I think I've got this working again. Come to find out, I had set up the IOC file to process each line individually instead of passing the file contents as a list 🤦

For S1 PQ - the fix for IOC file processing should also take care of the wacky definition file issue @xC0uNt3r7hr34t noted.

For S1 DV - after breaking the IOC file into chunks, they were getting merged back together due to the _process_queries() function. Code is set up to bypass that merging if the tag starts with the keyword "IOC".

For CbC - that issue stemmed from the base surveyor.py logic and no changes were needed in the product file.