Closed SteelBlueVision closed 6 years ago
Many thanks for rising the issue. These API's are exported only for IPsec usage when HMAC-SHAx is used for authentication and key size is larger than SHAx block size.
These API's are not really optimized for performance because these potential HMAC key reductions are not frequent operations in IPSec.
For best SHA hash performance and handling of large files I would recommend ISA-L crypto library https://github.com/01org/isa-l_crypto
Having said that we will have a look into this issue as the exported API should be functional for larger data sizes.
Thanks, Tomasz
Hi Tomasz, I have believe you mean to use isa-l_crypto since isa-l only hash CRC type hashing. Unfortunately, I am having an issue getting the correct SHA256 hash out of that as well. Please see the following issue report: https://github.com/01org/isa-l_crypto/issues/14
Correct, ISA-L crypto is the one that I pointed to. I tried to reproduce the problem using snippet of the code above (thanks!) and I couldn't find any mismatch between OpenSSL produced hashes vs ipsec-mb. Could you try to run again and help identify files that cause problems?
// sha256_test.cpp
#include <cstdint>
#include <fstream>
#include <intel-ipsec-mb.h>
#include <iomanip>
#include <iostream>
#include <string>
#define OPENSSL_NO_SHA1
#define OPENSSL_NO_SHA512
#include <openssl/sha.h>
static void
hexdump(FILE *fp,
const char *msg,
const void *p,
size_t len)
{
unsigned int i, out, ofs;
const unsigned char *data = (const unsigned char *) p;
fprintf(fp, "%s\n", msg);
ofs = 0;
while (ofs < len) {
char line[120];
out = snprintf(line, sizeof(line), "%08x:", ofs);
for (i = 0; ((ofs + i) < len) && (i < 16); i++)
out += snprintf(line + out, sizeof(line) - out,
" %02x", (data[ofs + i] & 0xff));
for (; i <= 16; i++)
out += snprintf(line + out, sizeof(line) - out, " | ");
for (i = 0; (ofs < len) && (i < 16); i++, ofs++) {
unsigned char c = data[ofs];
if ((c < ' ') || (c > '~'))
c = '.';
out += snprintf(line + out,
sizeof(line) - out, "%c", c);
}
fprintf(fp, "%s\n", line);
}
}
int main(int argc, char *argv[])
{
using namespace std;
std::cout << std::fixed;
if (argc!=3)
{
std::cerr << "Usage: file_to_be_hashed size_of_file_in_bytes\n";
return 1;
}
ifstream infile(argv[1]);
size_t BUF_SIZE=stoul(argv[2]);
char *buf=new char[BUF_SIZE];
std::clog << "Processing " << argv[1] << " of size: " << BUF_SIZE << '\n';
if (!infile.read(buf,BUF_SIZE))
{
std::cerr << "Could not read log file\n";
return 1;
}
infile.close();
uint8_t digest1[SHA256_DIGEST_SIZE_IN_BYTES]={0}; // 32-byte buffer
sha256_sse(buf,BUF_SIZE,digest1);
unsigned char digest2[SHA256_DIGEST_LENGTH];
SHA256_CTX sha256;
SHA256_Init(&sha256);
SHA256_Update(&sha256, buf, BUF_SIZE);
SHA256_Final(digest2, &sha256);
hexdump(stdout,"IPSEC-MB: ",digest1,sizeof(digest1));
hexdump(stdout,"OpenSSL: ",digest2,sizeof(digest2));
delete [] buf;
return 0;
}
Example output:
>./a.out ipsec_perf 75456
Processing ipsec_perf of size: 75456
IPSEC-MB:
00000000: db 8f 66 f8 72 8e d5 c6 62 a4 e7 97 2e d3 9a 7b | ..f.r...b......{
00000010: 3c 79 d0 04 03 9c b7 b9 12 4a a5 af a3 07 d2 c1 | <y.......J......
OpenSSL:
00000000: db 8f 66 f8 72 8e d5 c6 62 a4 e7 97 2e d3 9a 7b | ..f.r...b......{
00000010: 3c 79 d0 04 03 9c b7 b9 12 4a a5 af a3 07 d2 c1 | <y.......J......
Plus output for the test files mentioned above. Maybe this is something specific to sha256sum tool
>./a.out test-file.65536-bytes 65536
Processing test-file.65536-bytes of size: 65536
IPSEC-MB:
00000000: 34 dd 67 58 c2 90 8c f5 be 0e 41 11 73 f6 18 77 | 4.gX......A.s..w
00000010: f8 fa f2 dc dc fe d1 35 b5 a5 fa 90 57 74 47 bc | .......5....WtG.
OpenSSL:
00000000: 34 dd 67 58 c2 90 8c f5 be 0e 41 11 73 f6 18 77 | 4.gX......A.s..w
00000010: f8 fa f2 dc dc fe d1 35 b5 a5 fa 90 57 74 47 bc | .......5....WtG.
>./a.out test-file.65537-bytes 65537
Processing test-file.65537-bytes of size: 65537
IPSEC-MB:
00000000: 59 73 28 91 be bc c7 c6 74 d5 a0 e9 2e 0a 4e 04 | Ys(.....t.....N.
00000010: 08 f2 1d 9e 6c ce ca 1f ac e0 3c c1 ab 1c b6 7a | ....l.....<....z
OpenSSL:
00000000: 59 73 28 91 be bc c7 c6 74 d5 a0 e9 2e 0a 4e 04 | Ys(.....t.....N.
00000010: 08 f2 1d 9e 6c ce ca 1f ac e0 3c c1 ab 1c b6 7a | ....l.....<....z
I am closing the issue as ipsec-mb library and openssl produce same SHA256 hash values for the same buffers. Feel free to re-open if new/more evidence is available. Thanks, Tomasz
Note that this is true of all of the exported sha256_*() functions, not just the sse version. I am just using the sse version to demonstrate. I was hoping to use these functions to efficiently hash large files on an Intel based system, but if only blocks up to 64KB are supported for hashing, these functions are far less useful!
Here is a sample C++ program. You can try it on files that are <=65536 bytes in size and then on files that are >65536 bytes in size. It will fail to produce a correct hash for files that are >65536 bytes in size.
Examples:
Generate data files:
65536 byte example - SUCCESS
Output of application using sha256_sse() to compute the SHA256 hash:
Correct sha256sum output for comparison (matches above output at the hex level):
65537 byte example - FAILURE
Output of application using sha256_sse() to compute the SHA256 hash:
Correct sha256sum output for comparison (does not match above output at the hex level):
Source code: