Differences with the paper in the OpenSSL DES benchmarks

[Skepar]

Hello,

Thank you for your paper. I have been trying to reproduce the results you've obtained with Abacus, following the instructions specified in icse_artifact_evaluation.txt. Unfortunately, I have run into multiple issues.

It appears the script given in /scripts/start_base.sh fails to set the binaries' name correctly. This is due to the line bin_str=$(find /abacus/data/benchmarks/$alg_str-$lib_str-$ver_str/ -executable -type f), which considers files as executable according to their permissions, thus including many irrelevant files in data/benchmarks/. This causes Pin to fail when running, here for example running AES in mbedTLS 2.5:

cd /abacus/script/AES_MBEDTLS_2.5/
./start.sh
  Collecting Trace Files...
  E: Unable to load /abacus/Pintools/CryptoLibrary/obj-ia32/AES-mbedTLS-2.5.so. Check the architecture type.

However, Pin and Abacus both run as expected when following the instructions outlined at the beginning of the README. I've thus been able to reproduce most results as such, with two notable exceptions.

First, the DES benchmarks for OpenSSL 1.0.2f, 1.0.2k and 1.1.0f yield different results compared to the paper. Whereas Table II of the paper indicated 8 data-flow vulnerabilities for these versions, Abacus shows only 6. See below the output of Abacus for DES in OpenSSL 1.0.2f:

Total Constraints: 6
Control Transfer: 0
Data Access: 6
Information Leak for each address:
Address: 8048da3 Leaked:6.6 bits Type: DA  Num of Satisfied: 10 Total Sampling Numbers: 1000
Function Name:  DES_set_key_unchecked Module Name:  /abacus/data/benchmarks/DES-openssl-1.0.2f/des_ecb Offset: 307

Address: 8048daa Leaked:6.6 bits Type: DA  Num of Satisfied: 10 Total Sampling Numbers: 1000
Function Name:  DES_set_key_unchecked Module Name:  /abacus/data/benchmarks/DES-openssl-1.0.2f/des_ecb Offset: 314

Address: 8048db3 Leaked:8.5 bits Type: DA  Num of Satisfied: 11 Total Sampling Numbers: 4000
Function Name:  DES_set_key_unchecked Module Name:  /abacus/data/benchmarks/DES-openssl-1.0.2f/des_ecb Offset: 323

Address: 8048de0 Leaked:4.8 bits Type: DA  Num of Satisfied: 36 Total Sampling Numbers: 1000
Function Name:  DES_set_key_unchecked Module Name:  /abacus/data/benchmarks/DES-openssl-1.0.2f/des_ecb Offset: 368

Address: 8048df0 Leaked:2.0 bits Type: DA  Num of Satisfied: 251 Total Sampling Numbers: 1000
Function Name:  DES_set_key_unchecked Module Name:  /abacus/data/benchmarks/DES-openssl-1.0.2f/des_ecb Offset: 384

Address: 8048dfa Leaked:2.0 bits Type: DA  Num of Satisfied: 247 Total Sampling Numbers: 1000
Function Name:  DES_set_key_unchecked Module Name:  /abacus/data/benchmarks/DES-openssl-1.0.2f/des_ecb Offset: 394

inf
Time taken by SE: 0.2 seconds
Time taken by QIF: 5.4 seconds

Similar leaks are found for the 2 other versions mentioned, whether running QIF-old or QIF-new. I've been unsuccessful in trying to pinpoint where these differences could come from. Would you happen to have any lead on what could cause these differences?

I've also noticed that the benchmarks are missing build instructions, and the source code in the cases of poly1305 and Chacha20, which complicates such debugging process. It would be greatly helpful if you could push these on the repository.

Finally, I've been unable to reproduce the result for Ed25519 in Monocypher 3.0, as the associated Pintool outputs an empty instruction trace which can't be exploited by Abacus.

Setup: Artifacts retrieved from https://zenodo.org/record/4555757, Abacus was build with GCC 7.5 running in the supplied Docker image on Ubuntu 20.04.

Thank you for your time.

[qinkunbao]

Hello Skepar, thank you for using Abacus.

1. [DES] Could you try the benchmarks under the folder https://github.com/s3team/Abacus/tree/main/data/benchmarks Compiler optimizations may cause the difference.
2. [poly1305] and [Chacha20] (https://github.com/s3team/Abacus/tree/main/data/benchmarks) Here are benchmarks that contain all the debugging information and symbol information. You can easily get the source code by reverse engineering (e.g. IDA pro F5). BTW, Could you (@JesseMK ) please update the source code if possible?
3. Could you please run strings [ed25519_binary] | grep "crypto_key_exchange_public_key"? The Pintool will start to collect the traces once it executes the function.

[JesseMK]

Hello Skepar, thank you for using Abacus.

1. [DES] Could you try the benchmarks under the folder https://github.com/s3team/Abacus/tree/main/data/benchmarks Compiler optimizations may cause the difference.
2. [poly1305] and [Chacha20] (https://github.com/s3team/Abacus/tree/main/data/benchmarks) Here are benchmarks that contain all the debugging information and symbol information. You can easily get the source code by reverse engineering (e.g. IDA pro F5). BTW, Could you (@JesseMK ) please update the source code if possible?
3. Could you please run strings [ed25519_binary] | grep "crypto_key_exchange_public_key"? The Pintool will start to collect the traces once it executes the function.

2. Done. [poly1305] and [Chacha20] source code has been updated. https://github.com/s3team/Abacus/commit/f35c039796ef77a3eaba2b49ce3df7a27e17cca5

[qinkunbao]

Hello Skepar, thank you for using Abacus.

1. [DES] Could you try the benchmarks under the folder https://github.com/s3team/Abacus/tree/main/data/benchmarks Compiler optimizations may cause the difference.
2. [poly1305] and [Chacha20] (https://github.com/s3team/Abacus/tree/main/data/benchmarks) Here are benchmarks that contain all the debugging information and symbol information. You can easily get the source code by reverse engineering (e.g. IDA pro F5). BTW, Could you (@JesseMK ) please update the source code if possible?
3. Could you please run strings [ed25519_binary] | grep "crypto_key_exchange_public_key"? The Pintool will start to collect the traces once it executes the function.

2. Done. [poly1305] and [Chacha20] source code has been updated. https://github.com/s3team/Abacus/commit/f35c039796ef77a3eaba2b49ce3df7a27e17cca5

Thanks.

[Skepar]

Hello Skepar, thank you for using Abacus.

1. [DES] Could you try the benchmarks under the folder  https://github.com/s3team/Abacus/tree/main/data/benchmarks Compiler optimizations may cause the difference.

2. [poly1305] and [Chacha20] (https://github.com/s3team/Abacus/tree/main/data/benchmarks) Here are benchmarks that contain all the debugging information and symbol information. You can easily get the source code by reverse engineering (e.g. IDA pro F5).  BTW, Could you (@JesseMK ) please update the source code if possible?

3. Could you please run `strings [ed25519_binary]  | grep "crypto_key_exchange_public_key"`? The Pintool will start to collect the traces once it executes the function.

• I have run Abacus on the binaries supplied in https://github.com/s3team/Abacus/tree/main/data/benchmarks, wtihout recompiling any of them. I can confirm that for the DES-openssl benchmarks, Abacus finds 6 data access vulnerabilities instead of the 8 indicated in the paper for OpenSSL 1.0.2k, 1.0.2f and 1.1.0f.

• It seems there is no crypto_key_exchange_public_key symbol in this binary, however there is a crypto_x25519_public_key one. I modified the Pintool in Pintools/CryptoLibrary/ed25519-monocypher-3.0.cpp to target this function instead, and it produced a trace that could be exploited by Abacus (finding no vulnerabilities as indicated in the paper).