Open vsukhoml opened 3 months ago
This is a dupliate of #2111 and #19392.
Do we have dump of logs with entropy measurement somewhere? I see issues, but can't find how they are resolved.
NIST entropy tests produce output like below and shall be executed multiple times at different environmental points:
Loaded 1000000 samples of 2 distinct 1-bit-wide symbols
Running non-IID tests...
Running Most Common Value Estimate...
Literal MCV Estimate: mode = 500238, p-hat = 0.50023799999999996, p_u = 0.50152591514982692
Most Common Value Estimate = 0.995604 / 1 bit(s)
Running Entropic Statistic Estimates (bit strings only)...
Literal Collision Estimate: X-bar = 2.4995388341627653, sigma-hat = 0.50000041221288927, p = 0.53533662114698544
Collision Test Estimate = 0.901482 / 1 bit(s)
Literal Markov Estimate: P_0 = 0.49976199999999998, P_1 = 0.50023799999999996, P_0,0 = 0.5005132453312684, P_0,1 = 0.4994867546687316, P_1,0 = 0.49901047101579649, P_1,1 = 0.50098952898420346, p_max = 3.7793324099521518e-39
Markov Test Estimate = 0.997165 / 1 bit(s)
Literal Compression Estimate: X-bar = 5.2192586826314926, sigma-hat = 1.013874035750848, p = 0.027190262043963354
tCompression Test Estimate = 0.866794 / 1 bit(s)
Running Tuple Estimates...
Literal t-Tuple Estimate: t = 16, p-hat_max = 0.52755690708950331, p_u = 0.52884286485529719
Literal LRS Estimate: u = 17, v = 39, p-hat = 0.50128460831368271, p_u = 0.5025725193587246
T-Tuple Test Estimate = 0.919089 / 1 bit(s)
LRS Test Estimate = 0.992596 / 1 bit(s)
Running Predictor Estimates...
Literal MultiMCW Prediction Estimate: N = 999937, Pglobal' = 0.50123745267875108 (C = 499918) Plocal can't affect result (r = 19)
Multi Most Common in Window (MultiMCW) Prediction Test Estimate = 0.996434 / 1 bit(s)
Literal Lag Prediction Estimate: N = 999999, Pglobal' = 0.50050341356991546 (C = 499215) Plocal can't affect result (r = 20)
Lag Prediction Test Estimate = 0.998548 / 1 bit(s)
Literal MultiMMC Prediction Estimate: N = 999998, Pglobal' = 0.50190291683940447 (C = 500614) Plocal can't affect result (r = 17)
Multi Markov Model with Counting (MultiMMC) Prediction Test Estimate = 0.994520 / 1 bit(s)
Literal LZ78Y Prediction Estimate: N = 999983, Pglobal' = 0.50115842399842625 (C = 499862) Plocal can't affect result (r = 18)
LZ78Y Prediction Test Estimate = 0.996661 / 1 bit(s)
H_original: 0.866794
Minimal entropy 0.866794
Opening file: '/tmp/ea/trng_output_restart'
Loaded 1000000 samples made up of 2 distinct 1-bit-wide symbols.
H_I: 0.866794
ALPHA: 5.0251553006530614e-06, X_cutoff: 617
X_max: 569
Restart Sanity Check Passed...
Running non-IID tests...
Running Most Common Value Estimate...
Literal MCV Estimate: mode = 500768, p-hat = 0.50076799999999999, p_u = 0.50205591377644465
Most Common Value Estimate (Rows) = 0.994080 / 1 bit(s)
Literal MCV Estimate: mode = 500768, p-hat = 0.50076799999999999, p_u = 0.50205591377644465
Most Common Value Estimate (Cols) = 0.994080 / 1 bit(s)
Running Entropic Statistic Estimates (bit strings only)...
Literal Collision Estimate: X-bar = 2.4994313793611953, sigma-hat = 0.50000030152907682, p = 0.53608851711200045
Collision Test Estimate (Rows) = 0.899457 / 1 bit(s)
Literal Collision Estimate: X-bar = 2.4999850000749997, sigma-hat = 0.50000062477304874, p = 0.53202631140467849
Collision Test Estimate (Cols) = 0.910430 / 1 bit(s)
Literal Markov Estimate: P_0 = 0.50076799999999999, P_1 = 0.49923200000000001, P_0,0 = 0.50116820563614284, P_0,1 = 0.49883179436385716, P_1,0 = 0.50036756531545512, P_1,1 = 0.49963243468454488, p_max = 3.9586080948949002e-39
Markov Test Estimate (Rows) = 0.996642 / 1 bit(s)
Literal Markov Estimate: P_0 = 0.50076799999999999, P_1 = 0.49923200000000001, P_0,0 = 0.50150369033165054, P_0,1 = 0.49849630966834946, P_1,0 = 0.50003104775144169, P_1,1 = 0.49996895224855831, p_max = 4.309744037351884e-39
Markov Test Estimate (Cols) = 0.995684 / 1 bit(s)
Literal Compression Estimate: X-bar = 5.217106041939239, sigma-hat = 1.0163907231372382, p = 0.029793851221070922
Compression Test Estimate (Rows) = 0.844807 / 1 bit(s)
Literal Compression Estimate: X-bar = 5.2187618310457076, sigma-hat = 1.0139480373818941, p = 0.027823914855976351
Compression Test Estimate (Cols) = 0.861255 / 1 bit(s)
Running Tuple Estimates...
Literal t-Tuple Estimate: t = 15, p-hat_max = 0.51808453453604919, p_u = 0.51937160712968733
Literal LRS Estimate: u = 16, v = 41, p-hat = 0.51935546186299164, p_u = 0.52064241180346316
Literal t-Tuple Estimate: t = 15, p-hat_max = 0.52065048083654542, p_u = 0.52193729722018178
Literal LRS Estimate: u = 16, v = 39, p-hat = 0.50243644406537435, p_u = 0.5037243440702166
T-Tuple Test Estimate (Rows) = 0.945161 / 1 bit(s)
T-Tuple Test Estimate (Cols) = 0.938052 / 1 bit(s)
LRS Test Estimate (Rows) = 0.941635 / 1 bit(s)
LRS Test Estimate (Cols) = 0.989294 / 1 bit(s)
Running Predictor Estimates...
Literal MultiMCW Prediction Estimate: N = 999937, Pglobal' = 0.50171848251277584 (C = 500399) Plocal can't affect result (r = 21)
Multi Most Common in Window (MultiMCW) Prediction Test Estimate (Rows) = 0.995050 / 1 bit(s)
Literal MultiMCW Prediction Estimate: N = 999937, Pglobal' = 0.50108244281091086 (C = 499763) Plocal can't affect result (r = 22)
Multi Most Common in Window (MultiMCW) Prediction Test Estimate (Cols) = 0.996880 / 1 bit(s)
Literal Lag Prediction Estimate: N = 999999, Pglobal' = 0.50153641602912746 (C = 500248) Plocal can't affect result (r = 19)
Lag Prediction Test Estimate (Rows) = 0.995574 / 1 bit(s)
Literal Lag Prediction Estimate: N = 999999, Pglobal' = 0.50100441544916496 (C = 499716) Plocal can't affect result (r = 21)
Lag Prediction Test Estimate (Cols) = 0.997105 / 1 bit(s)
Literal MultiMMC Prediction Estimate: N = 999998, Pglobal' = 0.50168691697157541 (C = 500398) Plocal can't affect result (r = 20)
Multi Markov Model with Counting (MultiMMC) Prediction Test Estimate (Rows) = 0.995141 / 1 bit(s)
Literal MultiMMC Prediction Estimate: N = 999998, Pglobal' = 0.501786916940265 (C = 500498) Plocal can't affect result (r = 19)
Multi Markov Model with Counting (MultiMMC) Prediction Test Estimate (Cols) = 0.994853 / 1 bit(s)
Literal LZ78Y Prediction Estimate: N = 999983, Pglobal' = 0.50172843323195404 (C = 500432) Plocal can't affect result (r = 20)
LZ78Y Prediction Test Estimate (Rows) = 0.995021 / 1 bit(s)
Literal LZ78Y Prediction Estimate: N = 999983, Pglobal' = 0.50134142714530527 (C = 500045) Plocal can't affect result (r = 22)
LZ78Y Prediction Test Estimate (Cols) = 0.996135 / 1 bit(s)
H_r: 0.844807
H_c: 0.861255
H_I: 0.866794
Validation Test Passed...
min(H_r, H_c, H_I): 0.844807
Description
NIST 800-90B provides guidance how APT & RCT health checks shall be configured depending on required probability of detecting error and measured entropy. For the approved continuous health tests, the false positive probability $\alpha$ is recommended to be between $2^{−20}$ and $2^{−40}$. Lower probability values are acceptable. The submitter shall specify and document a false positive probability suitable for their application
AIS.31 defines parameters for other tests.
NIST Entropy Assessment toolkit includes set of test, documented in NIST 800-90B to measure entropy for use in configuring health checks appropritately.
In my experience measurements would be in the range ~0.8-0.85 per bit even for very high quality entropy source, or, say DRBG output due to compression test nuances on 1-bit streams (alphabet of entropy source).
So, action items are:
@vogelpi @jadephilipoom