Less blocking for basic functions

[malvidin]

For large files, the entropy, hashing , and base64 decoding would block. By adding an asyncio sleep after every 16MB, this blocking should be reduced.

[mlaferrera]

Thanks, @malvidin. Do you have any stats on the performance improvements for these changes? Theoretically I can see them being more performant, but I'd much rather know for sure before adding additional complexity.

[malvidin]

They are not more performant when compared directly to the non-blocking version when nothing else is running. I will try to get some tests done against multiple samples, instead of just testing the single functions.

I have been trying to get a single stoq run command to use all the cores, but so far I have not been successful.

Base64 Changes

1. Non-blocking
2. Takes twice as long (1.6 vs 0.7 seconds for 10 rounds in timeit)
3. Removes non-alphabet characters
4. Handles padding issues
5. The use of re.sub to clean the input appears to be the primary cause of slower response

Entropy Changes

1. Non-blocking
2. Not much difference (8.30 vs. 8.04 seconds for 5 rounds in timeit)

Hash Changes

1. Non-blocking
2. Not much difference (2.05 vs. 1.94 seconds for 10 rounds in timeit)
3. Easier to add/remove hash functions
4. Potential to use options to specify which hashes to return

ssdeep Changes

1. Non-blocking
2. Significantly slower (4.63 vs. 3.15 seconds for 10 rounds in timeit)
3. Mirrors the hash plugin

[mrisher23]

We have not forgotten about this pull request. Our apologies for the delay, but we are reviewing this as time permits.

[brbickel]

Close/reopen to force rebuild