Implement High-Quality Random Number Generation Using AES-CTR Mode with OpenSSL and AES-NI Support

[Knogle]

In this pull request, I present my implementation of a pseudo-random number generator (PRNG) utilizing the AES-CTR (Advanced Encryption Standard - Counter mode) in 128-bit mode. This implementation is designed to produce high-quality random numbers, which are essential for a wide range of cryptographic applications. By integrating with the OpenSSL library and exploiting AES-NI (Advanced Encryption Standard New Instructions) hardware acceleration when available, I ensure both the security and efficiency of the random number generation process. It provides the highest-quality of PRNGs yet for NWIPE, and is a CSPRNG.

Key Features:

AES-CTR Mode: I chose AES in Counter mode due to its renowned capability to generate secure and unpredictable pseudo-random sequences. This mode operates by encrypting incrementing counter values, with the encryption output serving as the stream of random bytes.

128-bit AES: Utilizing a 128-bit key size for AES encryption provides a strong security measure while maintaining efficient performance, adhering to current cryptographic standards for pseudo-random number generation.

Integration with OpenSSL: OpenSSL, being a well-established and rigorously tested cryptographic library, is used to manage AES operations. This integration ensures a high level of security and performance for the AES-CTR operations within our PRNG.

Leveraging AES-NI Support: My implementation automatically detects and utilizes AES-NI, a set of instructions that enhance AES operations on most modern processors. This feature significantly improves the speed of random number generation, reducing CPU usage and enhancing scalability.

Implementation Details:

Initialization: At the outset, the PRNG's state is initialized with a distinct 128-bit key and an initial counter value, using OpenSSL's AES_set_encrypt_key to prepare the AES key structure for subsequent operations.

Generating Random Numbers: For generating random numbers, the current counter value is encrypted under the configured AES key in CTR mode. The output of this encryption serves as the source of pseudo-random bytes, with the counter incremented after each operation to maintain the uniqueness of subsequent inputs.

State Management: The PRNG's internal state, including the AES key, counter (IV), and encryption buffer (ecount), is securely managed within an aes_ctr_state_t structure. This careful management is crucial for preserving the integrity and unpredictability of the random number stream.

Optimizing for Hardware: By optimizing for AES-NI, my implementation ensures enhanced performance through hardware acceleration, providing an efficient solution for generating random numbers across various applications.

This PRNG implementation stands as a robust and efficient tool for generating high-quality pseudo-random numbers, crucial for cryptographic operations, secure communications, and randomized algorithms. The combination of AES-CTR mode, OpenSSL's reliability, and the performance benefits of AES-NI hardware acceleration results in a superior random number generator.

I have ensured that the implementation is well-documented with clear comments, making it accessible for review, understanding, and maintenance, following best practices in both software development and cryptographic standards.

I look forward to receiving feedback on this pull request to further improve and ensure the effectiveness of the PRNG implementation.

Test of randomness:

• Mean frequency per byte value: Approximately 100,289
• Variance: Approximately 92,612
• Standard deviation: Approximately 304

NIST Test Suite:

A total of 188 tests (some of the 15 tests actually consist of multiple sub-tests)
were conducted to evaluate the randomness of 32 bitstreams of 1048576 bits from:

    /dev/loop0

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The numerous empirical results of these tests were then interpreted with
an examination of the proportion of sequences that pass a statistical test
(proportion analysis) and the distribution of p-values to check for uniformity
(uniformity analysis). The results were the following:

    188/188 tests passed successfully both the analyses.
    0/188 tests did not pass successfully both the analyses.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Here are the results of the single tests:

 - The "Frequency" test passed both the analyses.

 - The "Block Frequency" test passed both the analyses.

 - The "Cumulative Sums" (forward) test passed both the analyses.
   The "Cumulative Sums" (backward) test passed both the analyses.

 - The "Runs" test passed both the analyses.

 - The "Longest Run of Ones" test passed both the analyses.

 - The "Binary Matrix Rank" test passed both the analyses.

 - The "Discrete Fourier Transform" test passed both the analyses.

 - 148/148 of the "Non-overlapping Template Matching" tests passed both the analyses.

 - The "Overlapping Template Matching" test passed both the analyses.

 - The "Maurer's Universal Statistical" test passed both the analyses.

 - The "Approximate Entropy" test passed both the analyses.

 - 8/8 of the "Random Excursions" tests passed both the analyses.

 - 18/18 of the "Random Excursions Variant" tests passed both the analyses.

 - The "Serial" (first) test passed both the analyses.
   The "Serial" (second) test passed both the analyses.

 - The "Linear Complexity" test passed both the analyses.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The missing tests (if any) were whether disabled manually by the user or disabled
at run time due to input size requirements not satisfied by this run.

SmallCrush Test:

========= Summary results of SmallCrush =========

 Version:          TestU01 1.2.3
 Generator:        ufile_CreateReadBin
 Number of statistics:  15
 Total CPU time:   00:00:06.46

 All tests were passed

[PartialVolume]

I think so, i've created a 32-bit VM and was able to build the code there. But isn't it i386 then? Not sure tbh.

If you are just building nwipe then yes I would have thought it built as 32bit in the 32bit VM, what does uname -a return?. Should say i586 or i686 but not x86_64

I don't remember whether you are building ShredOS from source but this is how I build a 32 bit ShredOS including all the applications including nwipe. For testing 32 bit I generally build a 32 bit ShredOS and also build nwipe on a 32 bit distro.

For ShredOS you Just select the correct architecture and variant in menu config and you can build 32 bit on a x86_64 host.

The way I change the architecture and architecture variant is shown below from x86_64 variant nocona to i386 variant i586

Architecture options being:

ARC
ARC (big endian)
ARM (little endian)
ARM (big endian)  
AArch64 (little endian) 
AArch64 (big endian)
i386
m68k 
Microblaze AXI (little endian)
Microblaze non-AXI (big endian) 
MIPS (big endian)
MIPS (little endian)
MIPS64 (big endian)
MIPS64 (little endian)
OpenRISC
PowerPC 
PowerPC64 (big endian)
PowerPC64 (little endian)
RISCV 
s390x
SuperH
SPARC
SPARC64
x86_64 
Xtensa

x86_64 architecture variants are: ( I choose nocona for x86_64 so ShredOS runs on new and old processors back to Pentium 4 64bit ). There were Pentium 4 32bit and Pentium 4 64bit processors. I think all Intel processors before Pentium 4 were 32 bit.

x86-64
x86-64-v2
x86-64-v3
x86-64-v4
nocona
core2
corei7
nehalem
westmere
corei7-avx
sandybridge
ivybridge
haswell
broadwell
skylake
atom
bonnell
silvermont
goldmont
goldmont-plus
tremont
sierraforest
grandridge
knightslanding
knightsmill
skylake-avx512
cannonlake
icelake-client
icelake-server
cascadelake
cooperlake
tigerlake
sapphirerapids
alderlake
rocketlake
graniterapids
graniterapids-d
opteron
opteron w/ SSE3
barcelona
bobcat
jaguar
bulldozer
piledriver
steamroller
excavator
zen
zen 2
zen 3
zen 4

i386 (32 bit) variants: ( I select i586 which allows ShredOS to run on the first Pentium processors and everything after but not earlier processors like 486,386,286.

i486
i586
x1000
i686 pentium pro
pentium MMX
pentium mobile
pentium2
pentium3
pentium4 
prescott
nocona
core2
corei7
nehalem
westmere
corei7-avx
sandybridge
ivybridge
core-avx2
haswell
broadwell
skylake
atom
bonnell
silvermont
goldmont
goldmont-plus
tremont
sierraforest
grandridge
knightslanding
knightsmill
skylake-avx512
cannonlake
 icelake-client
cascadelake
cooperlake
tigerlake
sapphirerapids
alderlake
rocketlake
graniterapids
graniterapids-d
k6
k6-2
athlon
athlon-4
opteron
opteron w/ SSE3
barcelona
bobcat
jaguar
bulldozer
piledriver
steamroller
excavator
zen
zen 2
zen 3
zen 4
AMD Geode
Via/Cyrix C3 (Samuel/Ezra cores)
Via C3-2 (Nehemiah cores)
IDT Winchip C6

[Knogle]

Ahhhh alright, sound's great! Is there a way so i can try to build the current PR (this one) all together with ShredOS to test this?

I've build with Linux debian-i386 6.1.0-23-686-pae #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) i686 GNU/Linux

At least seems to run OK on i686.

But on these old CPUs without AES-Ni, Fibonacci is a lot lot faster.

[PartialVolume]

Yes, to build ShredOS with your modified version you need to do a release from your fork making sure you set the aes-ctr branch as the target. Then on your local copy of ShredOS You then need to edit a couple of files in packages/nwipe to change the sha1 hash to match your release and change the URL to point to your release. Then just rebuild ShredOS and it will pull in and compile your version of nwipe.

If you're unsure of anything let me know and I'll go into more detail.

[Knogle]

Yes, to build ShredOS with your modified version you need to do a release from your fork making sure you set the aes-ctr branch as the target. Then on your local copy of ShredOS You then need to edit a couple of files in packages/nwipe to change the sha1 hash to match your release and change the URL to point to your release. Then just rebuild ShredOS and it will pull in and compile your version of nwipe.

If you're unsure of anything let me know and I'll go into more detail.

Ahh thanks! Doing it this way, it worked really well now. Thanks for that. In the end i still forgot to edit the hash.

Currently wiping 4x 16TB drives in order to put them on eBay, running really well.

[Knogle]

Had to double check, but here we had the same issue. Depending on architecture the seed length was different, due to unsigned long, instead of uint64_t.

[PartialVolume]

@Knogle I've been thinking about whether to squash these 34 commits to a single commit, however I'm conflicted as to whether it's necessary or not in this case. Your commit comments are informative, however there are a few where you reverse a previous commit so the commit history would be tidier by squashing to a single commit.

I just wondered if you had a preference?

If I did squash the commits to a single commit would you want to do it in git and write the new commit comment for this branch or do you want me to do it in github by doing a merge squash and I write the new commit comment.?

[Knogle]

@Knogle I've been thinking about whether to squash these 34 commits to a single commit, however I'm conflicted as to whether it's necessary or not in this case. Your commit comments are informative, however there are a few where you reverse a previous commit so the commit history would be tidier by squashing to a single commit.

I just wondered if you had a preference?

If I did squash the commits to a single commit would you want to do it in git and write the new commit comment for this branch or do you want me to do it in github by doing a merge squash and I write the new commit comment.?

Hey, you could squash them by yourself if that's okay :)

[PartialVolume]

@Knogle I've been thinking about whether to squash these 34 commits to a single commit, however I'm conflicted as to whether it's necessary or not in this case. Your commit comments are informative, however there are a few where you reverse a previous commit so the commit history would be tidier by squashing to a single commit. I just wondered if you had a preference? If I did squash the commits to a single commit would you want to do it in git and write the new commit comment for this branch or do you want me to do it in github by doing a merge squash and I write the new commit comment.?

Hey, you could squash them by yourself if that's okay :)

yes, no problem.

[Knogle]

I think what's worth noting is 55472fb0e85ad5b9ea2ae9eed1f1b38f7508db8f, and fe493cfbbd76c3c8e03a7237138690d02682fe1e where AES-CTR is set as default option as well for AES-Ni enabled systems, otherwise falling back to Xoroshiro, and Lagged Fibonacci on i686.

[PartialVolume]

I think what's worth noting is 55472fb, where AES-CTR is set as default option as well for AES-Ni enabled systems, otherwise falling back to Xoroshiro, and Lagged Fibonacci on i686.

Noted, I will read through all the current commit comments and include information I think is important. My comment will probably lean towards being more verbose rather than succinct.

[PartialVolume]

@Knogle Can you also hold fire on producing any new branches based on your existing branch. I'm concerned I'm going to have quite a few merge conflicts to resolve when I come to merging your subsequent branches. So I'd like to get you existing work merged so you can then update your own fork before creating any new branches after all your existing PRs have been merged. Thanks.

[Knogle]

@Knogle Can you also hold fire on producing any new branches based on your existing branch. I'm concerned I'm going to have quite a few merge conflicts to resolve when I come to merging your subsequent branches. So I'd like to get you existing work merged so you can then update your own fork before creating any new branches after all your existing PRs have been merged. Thanks.

Sure, i will do so :)

[Knogle]

Currently conducting some tests on ARM :) Unfortunately my SD card is massively limiting.

[PartialVolume]

Yes, I've run 0.37 on arm, Ubuntu with xfce desktop on RPI-4 8GB RAM. Configured to boot via USB rather microsd. Seemed to work ok, I've not done any speed tests to see how fast a drive attached via USB will be subject to the drive limitations.

[Knogle]

@PartialVolume Maybe a consideration here, I currently have some time left so I can squash the commits. Regarding external libraries there are a few things we could do.

We could instead copy the relevant stuff out of the OpenSSL librarian and include it in our code without external dependency. The OpenSSL licence allows us to do so. Another approach which is more elegant. We include the stable OpenSSL version which works fine for as, as a submodule in the git project by specifying a specific release and tag/commit. This way we can have version locking, and always build with the same OpenSSL version, or altering the version only if we wish to do so. Second approach I have implemented in different projects already, and for different libraries, libmariadb in my case. Where I wanted the code to always function the same way.

Looks like this.

[PartialVolume]

Another approach which is more elegant. We include the stable OpenSSL version which works fine for as, as a submodule in the git project by specifying a specific release and tag/commit. This way we can have version locking, and always build with the same OpenSSL version, or altering the version only if we wish to do so. Second approach I have implemented in different projects already, and for different libraries, libmariadb in my case. Where I wanted the code to always function the same way.

Yes the second approach, that could work for us.

[Knogle]

Another approach which is more elegant. We include the stable OpenSSL version which works fine for as, as a submodule in the git project by specifying a specific release and tag/commit. This way we can have version locking, and always build with the same OpenSSL version, or altering the version only if we wish to do so. Second approach I have implemented in different projects already, and for different libraries, libmariadb in my case. Where I wanted the code to always function the same way.

Yes the second approach, that could work for us.

Ahoy, Squashed the commits, and also created a second approach here, using the submodules. https://github.com/martijnvanbrummelen/nwipe/pull/600