search

usnistgov / SP800-90B_EntropyAssessment

The SP800-90B_EntropyAssessment C++package implements the min-entropy assessment methods included in Special Publication 800-90B.
195 stars 88 forks source link

divsufsort64.h: No such file or directory #227

Closed hansonchar closed 9 months ago

hansonchar commented 9 months ago

On Amazon Linux, I've installed libdivsufsort:

$ git clone git@github.com:mpetri/FM-Index.git
$ cd FM-Index/libdivsufsort
$ mkdir build && cd build
$ cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_INSTALL_PREFIX="/usr/local" ..
$ make
$ sudo make install
[100%] Built target divsufsort
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/pkgconfig/libdivsufsort.pc
-- Installing: /usr/local/include/divsufsort.h
-- Installing: /usr/local/lib/libdivsufsort.a

But when I tried to make under SP800-90B_EntropyAssessment/cpp, I got the failure:

$ cd SP800-90B_EntropyAssessment/cpp
$ make
g++ -std=c++11 -fopenmp -O2 -ffloat-store -I/usr/include/jsoncpp -msse2 -march=native  iid_main.cpp -o ea_iid -lbz2 -lpthread -ldivsufsort -ldivsufsort64 -ljsoncpp -lcrypto
In file included from iid_main.cpp:6:0:
shared/lrs_test.h:6:10: fatal error: divsufsort64.h: No such file or directory
 #include <divsufsort64.h>
          ^~~~~~~~~~~~~~~~
compilation terminated.
make: *** [iid_main.o] Error 1

Any idea on how to fix this?

$ uname -a
Linux ip-172-31-45-108.ec2.internal 5.10.199-190.747.amzn2.x86_64 #1 SMP Sat Nov 4 16:55:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
hansonchar commented 9 months ago

Found a solution:

cmake -DCMAKE_BUILD_TYPE="Release" -DCMAKE_INSTALL_PREFIX="/usr/local" -DBUILD_DIVSUFSORT64="ON" ..
hansonchar commented 9 months ago

Also found this mentioned in the wiki:

To enable 64-bit libdivsufsort for large (>256MB) files, you may need to build libdivsufsort with BUILD_DIVSUFSORT64=YES.

joshuaehill commented 9 months ago

The 64-bit version of the divsufsort library became a dependency in the most recent version, after the merging of PR #226, which added support for large (>2GS) datasets. I'm glad that you figured out the issue and that @celic added the necessary hints in the Wiki.

I also do testing using Amazon Linux; on the most recent versions of Amazon Linux ($VERSION 2023 and later), I have to install cmake, gmp-devel, mpft-devel, zstd, gcc, gcc-c++, jasoncpp, jasoncpp-devel, openssl, and openssl-devel. The divsufsort library needs to be installed from source (as Amazon doesn't believe in providing a reasonable set of packages?)

Please feel free to note any additional problems that you run into; that said, I do a bunch of testing using EC2 instances, and it should work fine (given sufficient memory per process).

hansonchar commented 9 months ago

A different but related question. Trying to evaluate if a specific hardware satisfies the FIPS140-3 entropy requirements, I

However, I see messages like:

Can't open the provided file name: No such file or directory

Is there something I am missing?

More details below:

[jitterentropy-library/tests/raw-entropy/validation-runtime]$ ./processdata.sh
Building extractlsb ...
cc -pedantic -Wall -Wextra -O2   -c -o extractlsb.o extractlsb.c
cc extractlsb.o -o extractlsb  -lrt
Converting recorded entropy data ../results-measurements/jent-raw-noise-0001.data into different bit output
Processed 1000000 items from ./extractlsb samples with mask [0x000000000000000f] significant bits [4]
Constant 0s in var sample:
00000000 00000000 00000000 00000000 00000000 0000000- -------- --------
Constant 1s in var sample:
-------- -------- -------- -------- -------- -------- -------- --------
Constant 0s in single sample:
00000000 00000000 00000000 00000000 00000000 000000-- -------- --------
Constant 1s in single sample:
-------- -------- -------- -------- -------- -------- -------- --------
Processed 1000000 items from ./extractlsb samples with mask [0x00000000000000ff] significant bits [8]
Constant 0s in var sample:
00000000 00000000 00000000 00000000 00000000 0000000- -------- --------
Constant 1s in var sample:
-------- -------- -------- -------- -------- -------- -------- --------
Constant 0s in single sample:
00000000 00000000 00000000 00000000 00000000 000000-- -------- --------
Constant 1s in single sample:
-------- -------- -------- -------- -------- -------- -------- --------

Extraction finished. Now analyzing entropy for noise source ...

Analyzing entropy for ../results-analysis-runtime/jent-raw-noise-0001.0Fbitout.single.data 4-bit single
Analyzing entropy for ../results-analysis-runtime/jent-raw-noise-0001.0Fbitout.var.data 4-bit single
Analyzing entropy for ../results-analysis-runtime/jent-raw-noise-0001.FFbitout.single.data 8-bit single
Analyzing entropy for ../results-analysis-runtime/jent-raw-noise-0001.FFbitout.var.data 8-bit single

Now analyzing entropy for conditioned data...

Analyzing entropy for ../results-measurements/jent-conditioned.data 8-bit var
Can't open the provided file name: No such file or directory
joshuaehill commented 9 months ago

This doesn't seem to be related to the NIST tool. You're seeing scripting errors from the Jitter Entropy evaluation scripts, which I haven't really played with. You're probably best off asking at the Jitter Entropy Library GitHub page.

celic commented 9 months ago

Sorry, looks like more people are incorporating this project into automated pipelines. When I issue an update that breaks the build process, there may be some downwind effects. A good place to check is the Releases page. I'll try to be more explicit when something is changed that may require new libraries.

It may be helpful for automated pipelines to pull specific tags of this project rather than the master branch. This will ensure consistent usage and can be updated by the individual project managers when ready.

hansonchar commented 9 months ago

I think I figured it out. Details at https://github.com/smuellerDD/jitterentropy-library/issues/116.