search

usnistgov / SP800-90B_EntropyAssessment

The SP800-90B_EntropyAssessment C++package implements the min-entropy assessment methods included in Special Publication 800-90B.
202 stars 87 forks source link

LRS memory use for byte strings can be huge #53

Closed joshuaehill closed 5 years ago

joshuaehill commented 6 years ago

Non-pathological input files can lead to incredible memory consumption. If you look at the example data we published here: https://bit.ly/UL90BCOM Processing the "biased-random-bytes.bin" file consumes more than 3.5GB of memory. I'll retest on a larger scale instance and report back with the actual memory usage, but there's nothing about the test that absolutely requires a substantial amount of memory to be used. (Our internal tool uses 225MB for that data set...)

joshuaehill commented 6 years ago

It seems to be 5.3GB for that particular file.

joshuaehill commented 6 years ago

I fixed this in my reference branch: https://github.com/joshuaehill/SP800-90B_EntropyAssessment/tree/2018UL In this branch, I moved over to a SA/LCP array based implementation for the t-Tuple Estimate and LRS Estimate / Test.