search

besser82 / libxcrypt

Extended crypt library for descrypt, md5crypt, bcrypt, and others
GNU Lesser General Public License v2.1
194 stars 53 forks source link

yescrypt test fails on Linux/x86_64 systems configured with 1 GiB huge pages #152

Closed alexforencich closed 1 year ago

alexforencich commented 2 years ago

When building this package on an AMD EPYC Zen 2 CPU, the tests fail, including one reported segfault.

Configured with

  ./configure \
    --prefix=/usr \
    --disable-static \
    --enable-hashes=strong,glibc \
    --enable-obsolete-api=glibc \
    --disable-failure-tokens

Test suite run output:

PASS: test/ka-bcrypt
PASS: test/ka-bcrypt-a
SKIP: test/ka-bcrypt-x
PASS: test/ka-bcrypt-y
SKIP: test/ka-bigcrypt
SKIP: test/ka-bsdicrypt
PASS: test/ka-descrypt
PASS: test/ka-gost-yescrypt
PASS: test/ka-md5crypt
SKIP: test/ka-nt
PASS: test/ka-scrypt
SKIP: test/ka-sha1crypt
PASS: test/ka-sha256crypt
PASS: test/ka-sha512crypt
SKIP: test/ka-sunmd5
PASS: test/ka-yescrypt
PASS: test/alg-des
PASS: test/alg-gost3411-2012
PASS: test/alg-gost3411-2012-hmac
SKIP: test/alg-hmac-sha1
SKIP: test/alg-md4
PASS: test/alg-md5
PASS: test/alg-pbkdf-hmac-sha256
SKIP: test/alg-sha1
PASS: test/alg-sha256
PASS: test/alg-sha512
./build-aux/m4-autogen/test-driver: line 107: 669955 Segmentation fault      (core dumped) "$@" > $log_file 2>&1
FAIL: test/alg-yescrypt
PASS: test/badsalt
PASS: test/badsetting
PASS: test/byteorder
FAIL: test/checksalt
PASS: test/compile-strong-alias
PASS: test/crypt-badargs
PASS: test/crypt-gost-yescrypt
PASS: test/explicit-bzero
FAIL: test/gensalt
PASS: test/gensalt-extradata
SKIP: test/gensalt-nthash
PASS: test/getrandom-fallbacks
PASS: test/getrandom-interface
PASS: test/preferred-method
PASS: test/short-outbuf
PASS: test/special-char-salt
PASS: test/des-obsolete
PASS: test/des-obsolete_r
PASS: test/symbols-compat.pl
============================================================================
Testsuite summary for xcrypt 4.4.28
============================================================================
# TOTAL: 46
# PASS:  33
# SKIP:  10
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

test-suite.log:

=====================================
   xcrypt 4.4.28: ./test-suite.log
=====================================

# TOTAL: 46
# PASS:  33
# SKIP:  10
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

SKIP: test/ka-bcrypt-x
======================

SKIP test/ka-bcrypt-x (exit status: 77)

SKIP: test/ka-bigcrypt
======================

SKIP test/ka-bigcrypt (exit status: 77)

SKIP: test/ka-bsdicrypt
=======================

SKIP test/ka-bsdicrypt (exit status: 77)

SKIP: test/ka-nt
================

SKIP test/ka-nt (exit status: 77)

SKIP: test/ka-sha1crypt
=======================

SKIP test/ka-sha1crypt (exit status: 77)

SKIP: test/ka-sunmd5
====================

SKIP test/ka-sunmd5 (exit status: 77)

SKIP: test/alg-hmac-sha1
========================

SKIP test/alg-hmac-sha1 (exit status: 77)

SKIP: test/alg-md4
==================

SKIP test/alg-md4 (exit status: 77)

SKIP: test/alg-sha1
===================

SKIP test/alg-sha1 (exit status: 77)

FAIL: test/alg-yescrypt
=======================

scrypt("", "", 16, 1, 1) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
scrypt("password", "NaCl", 1024, 8, 16) = fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20
yescrypt("", "", 0, 4, 1, 1, 0, 0) = ef ad 0c 23 31 4c b5 72 bc 3c fb 15 43 da 42 f8 a8 b0 73 00 4c 86 6b 64 ab 50 55 a4 f0 9f a5 f5 71 14 2e bf e7 e0 5a 3b 92 c4 32 f3 1d ea 95 ad 5f 9c 85 4b 64 56 46 2f 4b d0 f7 32 b7 cd c5 49
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f 7f 1a e8 b4 df ed a5 1f 8b 6d 56 f3 08 1b e1 a7 c0 83 3b a2 71 9a 36 ab 02 88 5d ae 36 55 7d 34 26 86 b1 7b a7 5f 2c 21 77 92 de 09 70 ab 1d 07 a9 c7 50 93 6d 31 42 6f
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f
yescrypt("", "", 1, 4, 1, 1, 1, 0) = 4b aa 8c d8 60 8b a9 1f 3e 34 39 d9 ec 4f ae 8f 9f c0 92 d9 ca 22 b7 37 7e 31 ae 5b 9a d7 87 7c 11 68 69 11 62 dd 0e 5e f0 49 e5 70 65 0c be d4 38 4a d6 05 34 fb 0c be d1 9f f3 f0 33 c9 4b 0c
yescrypt("", "", 1, 4, 1, 1, 2, 0) = e6 e8 bb a0 9b 64 12 ff b0 b3 cc 35 e3 7d 0b 78 2a 47 fb aa dc 57 a0 76 d7 c6 cc 2e 70 91 9a 1b 8d 47 38 c4 f8 33 55 69 07 42 d9 be d7 1c 3b 8f b0 d7 eb 08 6a b1 34 c5 e5 57 07 c2 c1 3c 75 ef
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99 7d 70 ae 0a 64 bf 0a 4d 96 c1 73 ab f8 82 79 c1 a9 4a d9 bd f1 68 ed fb bd 90 f6 6e d5 c8 0d
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76 eb 24 1d f8 11 9a 9a 12 2a e3 69 20 bc c7 f4 14 b9 c0 d5 8f 45 00 80 60 da de 46 b0 c8 09 22 bd cc 16 a3 ab 5d 20 1d 4c 61 40 c6 71 be 1f 75 27 2c a9 04 73 9d 5a d1 ff 67 2b 0c 21
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68 ca 36 2c 55 57 cc 04 b6 81 1e 2e 73 08 41 f5 26 d8 f4 f7 ac fb fa 9e 06 fe 1f 38 3a 71 15 5e
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23
yescrypt("p", "s", 182, 16, 8, 1, 10, 0) = e1 f9 81 73 3a 94 05 2f cd 7a cb 14 05 df 0b bd e8 e4 99 b6 a1 33 1b 77 59 09 b4 8c 2f 51 6c 40 dc c8 30 16 35 b7 23 7b
yescrypt("p", "s", 1, 16, 8, 1, 10, 0) = 9e 7a 40 97 64 42 84 cf 3b 73 b6 04 50 ff 23 0c dc b6 b1 b1 9b 15 09 ee b4 82 f6 96 c4 f1 c7 05 c0 0f 74 02 16 18 3a 12
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99 c3 d6 69 30 fb 20 23 bb
yescrypt("p", "s", 1, 16, 8, 1, 0, 0) = 9d d6 36 c2 d0 bb 92 34 52 86 ef da f8 a6 8c fc 1b 4f fd c4 b1 ad ac cc 7d 86 4b 9a 67 87 b8 5d 6a e0 f5 28 0d a8 88 9f
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91
'$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.'
Plaintext: '(null)'
FAIL test/alg-yescrypt (exit status: 139)

FAIL: test/checksalt
====================

PASS (prefix): , result: 1
PASS (gensalt): 76, result: 3
PASS (crypt): 76tAku8b72z06, result: 3
PASS (prefix): .., result: 3
PASS (gensalt): 1H, result: 3
PASS (crypt): 1HkLTLEIfk./M, result: 3
PASS (prefix): MN, result: 3
PASS (gensalt): .E, result: 3
PASS (crypt): .EnlRW11z/Jsc, result: 3
PASS (prefix): _, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $1$, result: 3
PASS (gensalt): $1$32uG7C/O, result: 3
PASS (crypt): $1$32uG7C/O$Pgw7jYi84Y90VuzVFgRqm0, result: 3
PASS (prefix): $3$, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $md5, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $sha1, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $5$, result: 3
PASS (gensalt): $5$xmfhVe1sFEnk.9eI, result: 3
PASS (crypt): $5$xmfhVe1sFEnk.9eI$InZeA/rOVm3sZB6dZHHiNdmSrejta.H9Xrp/KmCOqG9, result: 3
PASS (prefix): $6$, result: 0
PASS (gensalt): $6$lahwDXRuUb9oDs.G, result: 0
PASS (crypt): $6$lahwDXRuUb9oDs.G$u.ODo/1GwDsQHxRpz0pArD/OJsH6OWki2oXHo6YdJbcuhi7VXfLg51DhyvP/hb.ikWVD8tRCFBSPEeLznxyMB0, result: 0
PASS (prefix): $2b$, result: 0
PASS (gensalt): $2b$05$Xsp3uErQ4uj9/HqQCs0Jce, result: 0
PASS (crypt): $2b$05$Xsp3uErQ4uj9/HqQCs0JceZwKLxO0Y4kNwbWe9s7d5fkS0fVNcn2a, result: 0
PASS (prefix): $2a$, result: 0
PASS (gensalt): $2a$05$Qbgz.buVJJXUKIXDmKeFZe, result: 0
PASS (crypt): $2a$05$Qbgz.buVJJXUKIXDmKeFZecDJ.weKbw6ManclBvnd.Y1mbaFApWTa, result: 0
PASS (prefix): $2x$, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $2y$, result: 0
PASS (gensalt): $2y$05$s90juXAZgFhJ2/S3mD95Q., result: 0
PASS (crypt): $2y$05$s90juXAZgFhJ2/S3mD95Q.KsIzdhN5M5wRgl3STkadyJSmj5w1ur2, result: 0
PASS (prefix): $y$, result: 0
PASS (gensalt): $y$j9T$T4gaPpCMTMRYH9HRjv3MD., result: 0
PASS (crypt): $y$j9T$T4gaPpCMTMRYH9HRjv3MD.$5VezkWWxSaISxFg9FXcWs9WsmCFfcsxlw5Gfd/6LEG7, result: 0
PASS (prefix): $7$, result: 0
PASS (gensalt): $7$CU..../....mqgW9OgdNu5T2rmh3CYb60, result: 0
FAIL (crypt): *0, expected: 0, got: 1
PASS (prefix): $gy$, result: 0
PASS (gensalt): $gy$j9T$Ph6gCkWZuTj/mkBMw0lUy1, result: 0
PASS (crypt): $gy$j9T$Ph6gCkWZuTj/mkBMw0lUy1$T9M/PoGChPjxSoLGBDSGruxquKbVeAZcGJ0QN0H0Vk3, result: 0
PASS (prefix): $@, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): %A, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): A%, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $2$, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): *0, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): *1, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix):   , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): !!, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): **, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ::, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ;;, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): \\, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix):   , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ��, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ��, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): (null), result: 1
PASS (gensalt): $y$j9T$/iLHwEwtradfwI06ytYZF1, result: 0
PASS (crypt): $y$j9T$/iLHwEwtradfwI06ytYZF1$86NUL9t5tShuBmLiOjBpz/pgzPKbAX.szRXNRDSsGhB, result: 0
FAIL test/checksalt (exit status: 1)

FAIL: test/gensalt
==================

   ok: /0/0 -> Mp
   ok: /0 -> crypt works with this salt
   ok: /0/1 -> Pp
   ok: /1 -> crypt works with this salt
   ok: /0/2 -> ZH
   ok: /2 -> crypt works with this salt
   ok: /0/3 -> Uh
   ok: /3 -> crypt works with this salt
   ok: /0/4 -> H1
   ok: /4 -> crypt works with this salt
   ok: $1$/0/0 -> $1$MJHnaAke
   ok: $1$/0 -> crypt works with this salt
   ok: $1$/0/1 -> $1$PKXc3hCO
   ok: $1$/1 -> crypt works with this salt
   ok: $1$/0/2 -> $1$ZAFlICwY
   ok: $1$/2 -> crypt works with this salt
   ok: $1$/0/3 -> $1$UqGBkVu0
   ok: $1$/3 -> crypt works with this salt
   ok: $1$/0/4 -> $1$FSjzyPMU
   ok: $1$/4 -> crypt works with this salt
   ok: $5$/0/0 -> $5$MJHnaAkegEVYHsFK
   ok: $5$/0 -> crypt works with this salt
   ok: $5$/0/1 -> $5$PKXc3hCOSyMqdaEQ
   ok: $5$/1 -> crypt works with this salt
   ok: $5$/0/2 -> $5$ZAFlICwYRETzIzIj
   ok: $5$/2 -> crypt works with this salt
   ok: $5$/0/3 -> $5$UqGBkVu01rurVZqg
   ok: $5$/3 -> crypt works with this salt
   ok: $5$/0/4 -> $5$quMzuPwIQXR6HQd/
   ok: $5$/4 -> crypt works with this salt
   ok: $5$/10191/0 -> $5$rounds=10191$MJHnaAkegEVYHsFK
   ok: $5$/10191/1 -> $5$rounds=10191$PKXc3hCOSyMqdaEQ
   ok: $5$/10191/2 -> $5$rounds=10191$ZAFlICwYRETzIzIj
   ok: $5$/10191/3 -> $5$rounds=10191$UqGBkVu01rurVZqg
   ok: $5$/10191/4 -> $5$rounds=10191$RnDYA.93uIpFFkA5
   ok: $5$/1/0 -> $5$rounds=1000$MJHnaAkegEVYHsFK
   ok: $5$/1/1 -> $5$rounds=1000$PKXc3hCOSyMqdaEQ
   ok: $5$/1/2 -> $5$rounds=1000$ZAFlICwYRETzIzIj
   ok: $5$/1/3 -> $5$rounds=1000$UqGBkVu01rurVZqg
   ok: $5$/1/4 -> $5$rounds=1000$/vuzxcWYlaOCG.P7
   ok: $5$/18446744073709551615/0 -> $5$rounds=999999999$MJHnaAkegEVYHsFK
   ok: $5$/18446744073709551615/1 -> $5$rounds=999999999$PKXc3hCOSyMqdaEQ
   ok: $5$/18446744073709551615/2 -> $5$rounds=999999999$ZAFlICwYRETzIzIj
   ok: $5$/18446744073709551615/3 -> $5$rounds=999999999$UqGBkVu01rurVZqg
   ok: $5$/18446744073709551615/4 -> $5$rounds=999999999$hxWgaRlF.dhwXBkY
   ok: $6$/0/0 -> $6$MJHnaAkegEVYHsFK
   ok: $6$/0 -> crypt works with this salt
   ok: $6$/0/1 -> $6$PKXc3hCOSyMqdaEQ
   ok: $6$/1 -> crypt works with this salt
   ok: $6$/0/2 -> $6$ZAFlICwYRETzIzIj
   ok: $6$/2 -> crypt works with this salt
   ok: $6$/0/3 -> $6$UqGBkVu01rurVZqg
   ok: $6$/3 -> crypt works with this salt
   ok: $6$/0/4 -> $6$F8t.UmlBc6BHlrPq
   ok: $6$/4 -> crypt works with this salt
   ok: $6$/10191/0 -> $6$rounds=10191$MJHnaAkegEVYHsFK
   ok: $6$/10191/1 -> $6$rounds=10191$PKXc3hCOSyMqdaEQ
   ok: $6$/10191/2 -> $6$rounds=10191$ZAFlICwYRETzIzIj
   ok: $6$/10191/3 -> $6$rounds=10191$UqGBkVu01rurVZqg
   ok: $6$/10191/4 -> $6$rounds=10191$Lf6VHVf86XhraO8C
   ok: $6$/1/0 -> $6$rounds=1000$MJHnaAkegEVYHsFK
   ok: $6$/1/1 -> $6$rounds=1000$PKXc3hCOSyMqdaEQ
   ok: $6$/1/2 -> $6$rounds=1000$ZAFlICwYRETzIzIj
   ok: $6$/1/3 -> $6$rounds=1000$UqGBkVu01rurVZqg
   ok: $6$/1/4 -> $6$rounds=1000$VuJL/wDmmTbrH9c9
   ok: $6$/18446744073709551615/0 -> $6$rounds=999999999$MJHnaAkegEVYHsFK
   ok: $6$/18446744073709551615/1 -> $6$rounds=999999999$PKXc3hCOSyMqdaEQ
   ok: $6$/18446744073709551615/2 -> $6$rounds=999999999$ZAFlICwYRETzIzIj
   ok: $6$/18446744073709551615/3 -> $6$rounds=999999999$UqGBkVu01rurVZqg
   ok: $6$/18446744073709551615/4 -> $6$rounds=999999999$4p2gf4MXDc7rgDrt
   ok: $2b$/0/0 -> $2b$05$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2b$/0 -> crypt works with this salt
   ok: $2b$/0/1 -> $2b$05$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2b$/1 -> crypt works with this salt
   ok: $2b$/0/2 -> $2b$05$HPNDjKMRFdR7zC87CMSmA.
   ok: $2b$/2 -> crypt works with this salt
   ok: $2b$/0/3 -> $2b$05$mAyzaIeJu41dWUkxEbn8hO
   ok: $2b$/3 -> crypt works with this salt
   ok: $2b$/0/4 -> $2b$05$SEaco.6srn7omzO.6BUmrO
   ok: $2b$/4 -> crypt works with this salt
   ok: $2b$/4/0 -> $2b$04$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2b$/4/1 -> $2b$04$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2b$/4/2 -> $2b$04$HPNDjKMRFdR7zC87CMSmA.
   ok: $2b$/4/3 -> $2b$04$mAyzaIeJu41dWUkxEbn8hO
   ok: $2b$/4/4 -> $2b$04$zW1EM59DJlMZtR8ztYE7c.
   ok: $2b$/31/0 -> $2b$31$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2b$/31/1 -> $2b$31$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2b$/31/2 -> $2b$31$HPNDjKMRFdR7zC87CMSmA.
   ok: $2b$/31/3 -> $2b$31$mAyzaIeJu41dWUkxEbn8hO
   ok: $2b$/31/4 -> $2b$31$.3BFgyI/oNwa.hZV8MAu4u
   ok: $2a$/0/0 -> $2a$05$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2a$/0 -> crypt works with this salt
   ok: $2a$/0/1 -> $2a$05$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2a$/1 -> crypt works with this salt
   ok: $2a$/0/2 -> $2a$05$HPNDjKMRFdR7zC87CMSmA.
   ok: $2a$/2 -> crypt works with this salt
   ok: $2a$/0/3 -> $2a$05$mAyzaIeJu41dWUkxEbn8hO
   ok: $2a$/3 -> crypt works with this salt
   ok: $2a$/0/4 -> $2a$05$nNN0wG2sqtPSMhVPmYY56e
   ok: $2a$/4 -> crypt works with this salt
   ok: $2y$/0/0 -> $2y$05$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2y$/0 -> crypt works with this salt
   ok: $2y$/0/1 -> $2y$05$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2y$/1 -> crypt works with this salt
   ok: $2y$/0/2 -> $2y$05$HPNDjKMRFdR7zC87CMSmA.
   ok: $2y$/2 -> crypt works with this salt
   ok: $2y$/0/3 -> $2y$05$mAyzaIeJu41dWUkxEbn8hO
   ok: $2y$/3 -> crypt works with this salt
   ok: $2y$/0/4 -> $2y$05$Wl1bzbtQk1UidWBumj1sIu
   ok: $2y$/4 -> crypt works with this salt
   ok: $y$/0/0 -> $y$j9T$MJHnaAkegEVYHsFKkmfzJ1
   ok: $y$/0 -> crypt works with this salt
   ok: $y$/0/1 -> $y$j9T$PKXc3hCOSyMqdaEQArI62/
   ok: $y$/1 -> crypt works with this salt
   ok: $y$/0/2 -> $y$j9T$ZAFlICwYRETzIzIjEIC86.
   ok: $y$/2 -> crypt works with this salt
   ok: $y$/0/3 -> $y$j9T$UqGBkVu01rurVZqgNchTB0
   ok: $y$/3 -> crypt works with this salt
   ok: $y$/0/4 -> $y$j9T$gzFASEx7j2QHHOsqMXODq.
   ok: $y$/4 -> crypt works with this salt
   ok: $y$/1/0 -> $y$j75$MJHnaAkegEVYHsFKkmfzJ1
   ok: $y$/1/1 -> $y$j75$PKXc3hCOSyMqdaEQArI62/
   ok: $y$/1/2 -> $y$j75$ZAFlICwYRETzIzIjEIC86.
   ok: $y$/1/3 -> $y$j75$UqGBkVu01rurVZqgNchTB0
   ok: $y$/1/4 -> $y$j75$3jP9OH1XY8xOUzDRctruS0
   ok: $y$/11/0 -> $y$jFT$MJHnaAkegEVYHsFKkmfzJ1
   ok: $y$/11/1 -> $y$jFT$PKXc3hCOSyMqdaEQArI62/
   ok: $y$/11/2 -> $y$jFT$ZAFlICwYRETzIzIjEIC86.
   ok: $y$/11/3 -> $y$jFT$UqGBkVu01rurVZqgNchTB0
   ok: $y$/11/4 -> $y$jFT$3W/upnTwMtYfZbMx.JdJF1
   ok: $7$/0/0 -> $7$CU..../....MJHnaAkegEVYHsFKkmfzJ1
ERROR: $7$/0 -> crypt(gensalt) fail
   ok: $7$/0/1 -> $7$CU..../....PKXc3hCOSyMqdaEQArI62/
ERROR: $7$/1 -> crypt(gensalt) fail
   ok: $7$/0/2 -> $7$CU..../....ZAFlICwYRETzIzIjEIC86.
ERROR: $7$/2 -> crypt(gensalt) fail
   ok: $7$/0/3 -> $7$CU..../....UqGBkVu01rurVZqgNchTB0
ERROR: $7$/3 -> crypt(gensalt) fail
   ok: $7$/0/4 -> $7$CU..../....go3Fzco9mLJVnaC.q9BEM/
ERROR: $7$/4 -> crypt(gensalt) fail
   ok: $7$/6/0 -> $7$BU..../....MJHnaAkegEVYHsFKkmfzJ1
   ok: $7$/6/1 -> $7$BU..../....PKXc3hCOSyMqdaEQArI62/
   ok: $7$/6/2 -> $7$BU..../....ZAFlICwYRETzIzIjEIC86.
   ok: $7$/6/3 -> $7$BU..../....UqGBkVu01rurVZqgNchTB0
   ok: $7$/6/4 -> $7$BU..../....ThjnUBaL9U795tMGR4GAz1
   ok: $7$/11/0 -> $7$GU..../....MJHnaAkegEVYHsFKkmfzJ1
   ok: $7$/11/1 -> $7$GU..../....PKXc3hCOSyMqdaEQArI62/
   ok: $7$/11/2 -> $7$GU..../....ZAFlICwYRETzIzIjEIC86.
   ok: $7$/11/3 -> $7$GU..../....UqGBkVu01rurVZqgNchTB0
   ok: $7$/11/4 -> $7$GU..../....UrDPoCOvFGRGFO1FZQwyF0
   ok: $gy$/0/0 -> $gy$j9T$MJHnaAkegEVYHsFKkmfzJ1
   ok: $gy$/0 -> crypt works with this salt
   ok: $gy$/0/1 -> $gy$j9T$PKXc3hCOSyMqdaEQArI62/
   ok: $gy$/1 -> crypt works with this salt
   ok: $gy$/0/2 -> $gy$j9T$ZAFlICwYRETzIzIjEIC86.
   ok: $gy$/2 -> crypt works with this salt
   ok: $gy$/0/3 -> $gy$j9T$UqGBkVu01rurVZqgNchTB0
   ok: $gy$/3 -> crypt works with this salt
   ok: $gy$/0/4 -> $gy$j9T$yBvb0ipsGXllFfLdOSwlM0
   ok: $gy$/4 -> crypt works with this salt
   ok: $gy$/1/0 -> $gy$j75$MJHnaAkegEVYHsFKkmfzJ1
   ok: $gy$/1/1 -> $gy$j75$PKXc3hCOSyMqdaEQArI62/
   ok: $gy$/1/2 -> $gy$j75$ZAFlICwYRETzIzIjEIC86.
   ok: $gy$/1/3 -> $gy$j75$UqGBkVu01rurVZqgNchTB0
   ok: $gy$/1/4 -> $gy$j75$c5gQCjcKiw32THQVv2.ZF1
   ok: $gy$/11/0 -> $gy$jFT$MJHnaAkegEVYHsFKkmfzJ1
   ok: $gy$/11/1 -> $gy$jFT$PKXc3hCOSyMqdaEQArI62/
   ok: $gy$/11/2 -> $gy$jFT$ZAFlICwYRETzIzIjEIC86.
   ok: $gy$/11/3 -> $gy$jFT$UqGBkVu01rurVZqgNchTB0
   ok: $gy$/11/4 -> $gy$jFT$5.0p4Ls7ZDB3Am0H5zZkZ1
FAIL test/gensalt (exit status: 1)

SKIP: test/gensalt-nthash
=========================

SKIP test/gensalt-nthash (exit status: 77)
thesamesam commented 2 years ago

Can you obtain a backtrace if you run it manually?

alexforencich commented 2 years ago

How do I run it manually?

solardiz commented 2 years ago

@alexforencich Can you try without --disable-failure-tokens?

Can you share more detail about the system - compiler, etc. Also is this in a VM, and if so with how much RAM?

solardiz commented 2 years ago

scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED

At least this one looks like an out-of-memory condition in a tiny VM to me.

alexforencich commented 2 years ago

I will re-run without that flag. This is not in a VM, this is bare metal on a 64 core AMD EPYC Zen 2 with 256 GB of RAM. So it's definitely not running out of RAM.

alexforencich commented 2 years ago

Seems like removing that flag made no difference to the test suite output:

=====================================
   xcrypt 4.4.28: ./test-suite.log
=====================================

# TOTAL: 46
# PASS:  33
# SKIP:  10
# XFAIL: 0
# FAIL:  3
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

SKIP: test/ka-bcrypt-x
======================

SKIP test/ka-bcrypt-x (exit status: 77)

SKIP: test/ka-bigcrypt
======================

SKIP test/ka-bigcrypt (exit status: 77)

SKIP: test/ka-bsdicrypt
=======================

SKIP test/ka-bsdicrypt (exit status: 77)

SKIP: test/ka-nt
================

SKIP test/ka-nt (exit status: 77)

SKIP: test/ka-sha1crypt
=======================

SKIP test/ka-sha1crypt (exit status: 77)

SKIP: test/ka-sunmd5
====================

SKIP test/ka-sunmd5 (exit status: 77)

SKIP: test/alg-hmac-sha1
========================

SKIP test/alg-hmac-sha1 (exit status: 77)

SKIP: test/alg-md4
==================

SKIP test/alg-md4 (exit status: 77)

SKIP: test/alg-sha1
===================

SKIP test/alg-sha1 (exit status: 77)

FAIL: test/alg-yescrypt
=======================

scrypt("", "", 16, 1, 1) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
scrypt("password", "NaCl", 1024, 8, 16) = fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20
yescrypt("", "", 0, 4, 1, 1, 0, 0) = ef ad 0c 23 31 4c b5 72 bc 3c fb 15 43 da 42 f8 a8 b0 73 00 4c 86 6b 64 ab 50 55 a4 f0 9f a5 f5 71 14 2e bf e7 e0 5a 3b 92 c4 32 f3 1d ea 95 ad 5f 9c 85 4b 64 56 46 2f 4b d0 f7 32 b7 cd c5 49
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f 7f 1a e8 b4 df ed a5 1f 8b 6d 56 f3 08 1b e1 a7 c0 83 3b a2 71 9a 36 ab 02 88 5d ae 36 55 7d 34 26 86 b1 7b a7 5f 2c 21 77 92 de 09 70 ab 1d 07 a9 c7 50 93 6d 31 42 6f
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f
yescrypt("", "", 1, 4, 1, 1, 1, 0) = 4b aa 8c d8 60 8b a9 1f 3e 34 39 d9 ec 4f ae 8f 9f c0 92 d9 ca 22 b7 37 7e 31 ae 5b 9a d7 87 7c 11 68 69 11 62 dd 0e 5e f0 49 e5 70 65 0c be d4 38 4a d6 05 34 fb 0c be d1 9f f3 f0 33 c9 4b 0c
yescrypt("", "", 1, 4, 1, 1, 2, 0) = e6 e8 bb a0 9b 64 12 ff b0 b3 cc 35 e3 7d 0b 78 2a 47 fb aa dc 57 a0 76 d7 c6 cc 2e 70 91 9a 1b 8d 47 38 c4 f8 33 55 69 07 42 d9 be d7 1c 3b 8f b0 d7 eb 08 6a b1 34 c5 e5 57 07 c2 c1 3c 75 ef
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99 7d 70 ae 0a 64 bf 0a 4d 96 c1 73 ab f8 82 79 c1 a9 4a d9 bd f1 68 ed fb bd 90 f6 6e d5 c8 0d
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76 eb 24 1d f8 11 9a 9a 12 2a e3 69 20 bc c7 f4 14 b9 c0 d5 8f 45 00 80 60 da de 46 b0 c8 09 22 bd cc 16 a3 ab 5d 20 1d 4c 61 40 c6 71 be 1f 75 27 2c a9 04 73 9d 5a d1 ff 67 2b 0c 21
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68 ca 36 2c 55 57 cc 04 b6 81 1e 2e 73 08 41 f5 26 d8 f4 f7 ac fb fa 9e 06 fe 1f 38 3a 71 15 5e
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23
yescrypt("p", "s", 182, 16, 8, 1, 10, 0) = e1 f9 81 73 3a 94 05 2f cd 7a cb 14 05 df 0b bd e8 e4 99 b6 a1 33 1b 77 59 09 b4 8c 2f 51 6c 40 dc c8 30 16 35 b7 23 7b
yescrypt("p", "s", 1, 16, 8, 1, 10, 0) = 9e 7a 40 97 64 42 84 cf 3b 73 b6 04 50 ff 23 0c dc b6 b1 b1 9b 15 09 ee b4 82 f6 96 c4 f1 c7 05 c0 0f 74 02 16 18 3a 12
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99 c3 d6 69 30 fb 20 23 bb
yescrypt("p", "s", 1, 16, 8, 1, 0, 0) = 9d d6 36 c2 d0 bb 92 34 52 86 ef da f8 a6 8c fc 1b 4f fd c4 b1 ad ac cc 7d 86 4b 9a 67 87 b8 5d 6a e0 f5 28 0d a8 88 9f
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91
'$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.'
Plaintext: '(null)'
FAIL test/alg-yescrypt (exit status: 139)

FAIL: test/checksalt
====================

PASS (prefix): , result: 1
PASS (gensalt): eh, result: 3
PASS (crypt): ehop.Wd4MXRDk, result: 3
PASS (prefix): .., result: 3
PASS (gensalt): sY, result: 3
PASS (crypt): sY5yosB26SRwg, result: 3
PASS (prefix): MN, result: 3
PASS (gensalt): im, result: 3
PASS (crypt): imXvtWPvxc/sA, result: 3
PASS (prefix): _, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $1$, result: 3
PASS (gensalt): $1$xzxRkDGL, result: 3
PASS (crypt): $1$xzxRkDGL$1jclgODNAycDFwa1wVAPM1, result: 3
PASS (prefix): $3$, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $md5, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $sha1, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $5$, result: 3
PASS (gensalt): $5$IeBcVJPHRO2Yk85s, result: 3
PASS (crypt): $5$IeBcVJPHRO2Yk85s$.FjIdLvxnPH692/fAmA5uAYrIalGwRgLV3Y15ltj436, result: 3
PASS (prefix): $6$, result: 0
PASS (gensalt): $6$xv2V0kwnZpTYSBWU, result: 0
PASS (crypt): $6$xv2V0kwnZpTYSBWU$h6v.i/E17NUOv6At9.hIXp7lFn55.51aOTvWq6jRHPJsoQn6k/a.5nlNiD87Y2Qizt69ggFv4EXlAxQLBBal.1, result: 0
PASS (prefix): $2b$, result: 0
PASS (gensalt): $2b$05$/yjrVBI7jbLXSuwZpGzWze, result: 0
PASS (crypt): $2b$05$/yjrVBI7jbLXSuwZpGzWzeUbZz06J5KOBFoYzwPpR4KTK0lIiG15W, result: 0
PASS (prefix): $2a$, result: 0
PASS (gensalt): $2a$05$eTvz1eSdeFS37MZgKRnCFO, result: 0
PASS (crypt): $2a$05$eTvz1eSdeFS37MZgKRnCFOAAn1V89Ui/OVSzsVpyPsHNSnFKnymu2, result: 0
PASS (prefix): $2x$, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $2y$, result: 0
PASS (gensalt): $2y$05$NKZYLT.MhLuzlPLAJNly4O, result: 0
PASS (crypt): $2y$05$NKZYLT.MhLuzlPLAJNly4OUNq3xsTivE/nyAmfXgz3xxvFBx4WsWO, result: 0
PASS (prefix): $y$, result: 0
PASS (gensalt): $y$j9T$J55i.8.DyohCLFykZJJ4V1, result: 0
PASS (crypt): $y$j9T$J55i.8.DyohCLFykZJJ4V1$IVnfOeBy7MwWlSq23EOTBicdFogUfTbZ0EF7.3uBvDB, result: 0
PASS (prefix): $7$, result: 0
PASS (gensalt): $7$CU..../....5JzRc6wzrecHvOnBnhbop., result: 0
FAIL (crypt): *0, expected: 0, got: 1
PASS (prefix): $gy$, result: 0
PASS (gensalt): $gy$j9T$IBzDUyD/E0zkZwsbalwsm1, result: 0
PASS (crypt): $gy$j9T$IBzDUyD/E0zkZwsbalwsm1$KZIgvtuBV8F04hkbH6ZcCFuZX9eeK5linCJvrVWXNo5, result: 0
PASS (prefix): $@, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): %A, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): A%, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): $2$, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): *0, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): *1, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix):   , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): !!, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): **, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ::, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ;;, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): \\, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix):   , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): , result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ��, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): ��, result: 1
PASS (gensalt): *0, result: 1
PASS (crypt): *1, result: 1
PASS (prefix): (null), result: 1
PASS (gensalt): $y$j9T$TpTYh45dZeZaXQQiiKFPl., result: 0
PASS (crypt): $y$j9T$TpTYh45dZeZaXQQiiKFPl.$Dsu4FGloZU5K5POM8dVM3T27HqHOItpS8OPwHwyiwZ., result: 0
FAIL test/checksalt (exit status: 1)

FAIL: test/gensalt
==================

   ok: /0/0 -> Mp
   ok: /0 -> crypt works with this salt
   ok: /0/1 -> Pp
   ok: /1 -> crypt works with this salt
   ok: /0/2 -> ZH
   ok: /2 -> crypt works with this salt
   ok: /0/3 -> Uh
   ok: /3 -> crypt works with this salt
   ok: /0/4 -> bs
   ok: /4 -> crypt works with this salt
   ok: $1$/0/0 -> $1$MJHnaAke
   ok: $1$/0 -> crypt works with this salt
   ok: $1$/0/1 -> $1$PKXc3hCO
   ok: $1$/1 -> crypt works with this salt
   ok: $1$/0/2 -> $1$ZAFlICwY
   ok: $1$/2 -> crypt works with this salt
   ok: $1$/0/3 -> $1$UqGBkVu0
   ok: $1$/3 -> crypt works with this salt
   ok: $1$/0/4 -> $1$zGfOVjXy
   ok: $1$/4 -> crypt works with this salt
   ok: $5$/0/0 -> $5$MJHnaAkegEVYHsFK
   ok: $5$/0 -> crypt works with this salt
   ok: $5$/0/1 -> $5$PKXc3hCOSyMqdaEQ
   ok: $5$/1 -> crypt works with this salt
   ok: $5$/0/2 -> $5$ZAFlICwYRETzIzIj
   ok: $5$/2 -> crypt works with this salt
   ok: $5$/0/3 -> $5$UqGBkVu01rurVZqg
   ok: $5$/3 -> crypt works with this salt
   ok: $5$/0/4 -> $5$1xL/I9AJn3MZxlR7
   ok: $5$/4 -> crypt works with this salt
   ok: $5$/10191/0 -> $5$rounds=10191$MJHnaAkegEVYHsFK
   ok: $5$/10191/1 -> $5$rounds=10191$PKXc3hCOSyMqdaEQ
   ok: $5$/10191/2 -> $5$rounds=10191$ZAFlICwYRETzIzIj
   ok: $5$/10191/3 -> $5$rounds=10191$UqGBkVu01rurVZqg
   ok: $5$/10191/4 -> $5$rounds=10191$qD5/mHNGuRlL3ncc
   ok: $5$/1/0 -> $5$rounds=1000$MJHnaAkegEVYHsFK
   ok: $5$/1/1 -> $5$rounds=1000$PKXc3hCOSyMqdaEQ
   ok: $5$/1/2 -> $5$rounds=1000$ZAFlICwYRETzIzIj
   ok: $5$/1/3 -> $5$rounds=1000$UqGBkVu01rurVZqg
   ok: $5$/1/4 -> $5$rounds=1000$dRlX8Fm5d8HdAAtN
   ok: $5$/18446744073709551615/0 -> $5$rounds=999999999$MJHnaAkegEVYHsFK
   ok: $5$/18446744073709551615/1 -> $5$rounds=999999999$PKXc3hCOSyMqdaEQ
   ok: $5$/18446744073709551615/2 -> $5$rounds=999999999$ZAFlICwYRETzIzIj
   ok: $5$/18446744073709551615/3 -> $5$rounds=999999999$UqGBkVu01rurVZqg
   ok: $5$/18446744073709551615/4 -> $5$rounds=999999999$LRUpxVzIPcXtEn80
   ok: $6$/0/0 -> $6$MJHnaAkegEVYHsFK
   ok: $6$/0 -> crypt works with this salt
   ok: $6$/0/1 -> $6$PKXc3hCOSyMqdaEQ
   ok: $6$/1 -> crypt works with this salt
   ok: $6$/0/2 -> $6$ZAFlICwYRETzIzIj
   ok: $6$/2 -> crypt works with this salt
   ok: $6$/0/3 -> $6$UqGBkVu01rurVZqg
   ok: $6$/3 -> crypt works with this salt
   ok: $6$/0/4 -> $6$rN1pi6c9tESBt.C4
   ok: $6$/4 -> crypt works with this salt
   ok: $6$/10191/0 -> $6$rounds=10191$MJHnaAkegEVYHsFK
   ok: $6$/10191/1 -> $6$rounds=10191$PKXc3hCOSyMqdaEQ
   ok: $6$/10191/2 -> $6$rounds=10191$ZAFlICwYRETzIzIj
   ok: $6$/10191/3 -> $6$rounds=10191$UqGBkVu01rurVZqg
   ok: $6$/10191/4 -> $6$rounds=10191$QQLo.8Kd98TvFmbv
   ok: $6$/1/0 -> $6$rounds=1000$MJHnaAkegEVYHsFK
   ok: $6$/1/1 -> $6$rounds=1000$PKXc3hCOSyMqdaEQ
   ok: $6$/1/2 -> $6$rounds=1000$ZAFlICwYRETzIzIj
   ok: $6$/1/3 -> $6$rounds=1000$UqGBkVu01rurVZqg
   ok: $6$/1/4 -> $6$rounds=1000$TRzQU1jZ0kHR.ksn
   ok: $6$/18446744073709551615/0 -> $6$rounds=999999999$MJHnaAkegEVYHsFK
   ok: $6$/18446744073709551615/1 -> $6$rounds=999999999$PKXc3hCOSyMqdaEQ
   ok: $6$/18446744073709551615/2 -> $6$rounds=999999999$ZAFlICwYRETzIzIj
   ok: $6$/18446744073709551615/3 -> $6$rounds=999999999$UqGBkVu01rurVZqg
   ok: $6$/18446744073709551615/4 -> $6$rounds=999999999$EtByMjE9bc1xb4ad
   ok: $2b$/0/0 -> $2b$05$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2b$/0 -> crypt works with this salt
   ok: $2b$/0/1 -> $2b$05$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2b$/1 -> crypt works with this salt
   ok: $2b$/0/2 -> $2b$05$HPNDjKMRFdR7zC87CMSmA.
   ok: $2b$/2 -> crypt works with this salt
   ok: $2b$/0/3 -> $2b$05$mAyzaIeJu41dWUkxEbn8hO
   ok: $2b$/3 -> crypt works with this salt
   ok: $2b$/0/4 -> $2b$05$h6Bbwk1mcM3SujHUpJM7Ce
   ok: $2b$/4 -> crypt works with this salt
   ok: $2b$/4/0 -> $2b$04$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2b$/4/1 -> $2b$04$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2b$/4/2 -> $2b$04$HPNDjKMRFdR7zC87CMSmA.
   ok: $2b$/4/3 -> $2b$04$mAyzaIeJu41dWUkxEbn8hO
   ok: $2b$/4/4 -> $2b$04$oiwrEARn.V8nqRkbaeWYou
   ok: $2b$/31/0 -> $2b$31$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2b$/31/1 -> $2b$31$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2b$/31/2 -> $2b$31$HPNDjKMRFdR7zC87CMSmA.
   ok: $2b$/31/3 -> $2b$31$mAyzaIeJu41dWUkxEbn8hO
   ok: $2b$/31/4 -> $2b$31$ovS.ihwymi50nzxELbxMD.
   ok: $2a$/0/0 -> $2a$05$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2a$/0 -> crypt works with this salt
   ok: $2a$/0/1 -> $2a$05$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2a$/1 -> crypt works with this salt
   ok: $2a$/0/2 -> $2a$05$HPNDjKMRFdR7zC87CMSmA.
   ok: $2a$/2 -> crypt works with this salt
   ok: $2a$/0/3 -> $2a$05$mAyzaIeJu41dWUkxEbn8hO
   ok: $2a$/3 -> crypt works with this salt
   ok: $2a$/0/4 -> $2a$05$pFIrMVv6Ta9e/iKIWqB7b.
   ok: $2a$/4 -> crypt works with this salt
   ok: $2y$/0/0 -> $2y$05$UBVLHeMpJ/QQCv3XqJx8zO
   ok: $2y$/0 -> crypt works with this salt
   ok: $2y$/0/1 -> $2y$05$kxUgPcrmlm9XoOjvxCyfP.
   ok: $2y$/1 -> crypt works with this salt
   ok: $2y$/0/2 -> $2y$05$HPNDjKMRFdR7zC87CMSmA.
   ok: $2y$/2 -> crypt works with this salt
   ok: $2y$/0/3 -> $2y$05$mAyzaIeJu41dWUkxEbn8hO
   ok: $2y$/3 -> crypt works with this salt
   ok: $2y$/0/4 -> $2y$05$mjhF7cZEMrIM2S.rhN21ae
   ok: $2y$/4 -> crypt works with this salt
   ok: $y$/0/0 -> $y$j9T$MJHnaAkegEVYHsFKkmfzJ1
   ok: $y$/0 -> crypt works with this salt
   ok: $y$/0/1 -> $y$j9T$PKXc3hCOSyMqdaEQArI62/
   ok: $y$/1 -> crypt works with this salt
   ok: $y$/0/2 -> $y$j9T$ZAFlICwYRETzIzIjEIC86.
   ok: $y$/2 -> crypt works with this salt
   ok: $y$/0/3 -> $y$j9T$UqGBkVu01rurVZqgNchTB0
   ok: $y$/3 -> crypt works with this salt
   ok: $y$/0/4 -> $y$j9T$C8Sq6MX47Q94siiqw70zo0
   ok: $y$/4 -> crypt works with this salt
   ok: $y$/1/0 -> $y$j75$MJHnaAkegEVYHsFKkmfzJ1
   ok: $y$/1/1 -> $y$j75$PKXc3hCOSyMqdaEQArI62/
   ok: $y$/1/2 -> $y$j75$ZAFlICwYRETzIzIjEIC86.
   ok: $y$/1/3 -> $y$j75$UqGBkVu01rurVZqgNchTB0
   ok: $y$/1/4 -> $y$j75$Lu1Q9/EhDUczWmR7doInW.
   ok: $y$/11/0 -> $y$jFT$MJHnaAkegEVYHsFKkmfzJ1
   ok: $y$/11/1 -> $y$jFT$PKXc3hCOSyMqdaEQArI62/
   ok: $y$/11/2 -> $y$jFT$ZAFlICwYRETzIzIjEIC86.
   ok: $y$/11/3 -> $y$jFT$UqGBkVu01rurVZqgNchTB0
   ok: $y$/11/4 -> $y$jFT$RahsS09D6zSyKkNMpk84V1
   ok: $7$/0/0 -> $7$CU..../....MJHnaAkegEVYHsFKkmfzJ1
ERROR: $7$/0 -> crypt(gensalt) fail
   ok: $7$/0/1 -> $7$CU..../....PKXc3hCOSyMqdaEQArI62/
ERROR: $7$/1 -> crypt(gensalt) fail
   ok: $7$/0/2 -> $7$CU..../....ZAFlICwYRETzIzIjEIC86.
ERROR: $7$/2 -> crypt(gensalt) fail
   ok: $7$/0/3 -> $7$CU..../....UqGBkVu01rurVZqgNchTB0
ERROR: $7$/3 -> crypt(gensalt) fail
   ok: $7$/0/4 -> $7$CU..../....r8lXdefs2sucjXQNRom45.
ERROR: $7$/4 -> crypt(gensalt) fail
   ok: $7$/6/0 -> $7$BU..../....MJHnaAkegEVYHsFKkmfzJ1
   ok: $7$/6/1 -> $7$BU..../....PKXc3hCOSyMqdaEQArI62/
   ok: $7$/6/2 -> $7$BU..../....ZAFlICwYRETzIzIjEIC86.
   ok: $7$/6/3 -> $7$BU..../....UqGBkVu01rurVZqgNchTB0
   ok: $7$/6/4 -> $7$BU..../.....8cAFendN2evCS2psKSRr.
   ok: $7$/11/0 -> $7$GU..../....MJHnaAkegEVYHsFKkmfzJ1
   ok: $7$/11/1 -> $7$GU..../....PKXc3hCOSyMqdaEQArI62/
   ok: $7$/11/2 -> $7$GU..../....ZAFlICwYRETzIzIjEIC86.
   ok: $7$/11/3 -> $7$GU..../....UqGBkVu01rurVZqgNchTB0
   ok: $7$/11/4 -> $7$GU..../....zgJ2ooaS4sb3Otkq.hThd0
   ok: $gy$/0/0 -> $gy$j9T$MJHnaAkegEVYHsFKkmfzJ1
   ok: $gy$/0 -> crypt works with this salt
   ok: $gy$/0/1 -> $gy$j9T$PKXc3hCOSyMqdaEQArI62/
   ok: $gy$/1 -> crypt works with this salt
   ok: $gy$/0/2 -> $gy$j9T$ZAFlICwYRETzIzIjEIC86.
   ok: $gy$/2 -> crypt works with this salt
   ok: $gy$/0/3 -> $gy$j9T$UqGBkVu01rurVZqgNchTB0
   ok: $gy$/3 -> crypt works with this salt
   ok: $gy$/0/4 -> $gy$j9T$5H4lj9ZYcPyDAhom15mFe1
   ok: $gy$/4 -> crypt works with this salt
   ok: $gy$/1/0 -> $gy$j75$MJHnaAkegEVYHsFKkmfzJ1
   ok: $gy$/1/1 -> $gy$j75$PKXc3hCOSyMqdaEQArI62/
   ok: $gy$/1/2 -> $gy$j75$ZAFlICwYRETzIzIjEIC86.
   ok: $gy$/1/3 -> $gy$j75$UqGBkVu01rurVZqgNchTB0
   ok: $gy$/1/4 -> $gy$j75$HcKJ545e5x.sfKKxqbIeL1
   ok: $gy$/11/0 -> $gy$jFT$MJHnaAkegEVYHsFKkmfzJ1
   ok: $gy$/11/1 -> $gy$jFT$PKXc3hCOSyMqdaEQArI62/
   ok: $gy$/11/2 -> $gy$jFT$ZAFlICwYRETzIzIjEIC86.
   ok: $gy$/11/3 -> $gy$jFT$UqGBkVu01rurVZqgNchTB0
   ok: $gy$/11/4 -> $gy$jFT$7BnuVoJLQ1yO5eAoYbida/
FAIL test/gensalt (exit status: 1)

SKIP: test/gensalt-nthash
=========================

SKIP test/gensalt-nthash (exit status: 77)
alexforencich commented 2 years ago

Some system information:

$ uname -a
Linux frodo 5.16.9-arch1-1 #1 SMP PREEMPT Fri, 11 Feb 2022 22:42:06 +0000 x86_64 GNU/Linux
$ gcc --version
gcc (GCC) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Build works just fine on a machine with an Intel CPU, with the same OS and compiler versions.

solardiz commented 2 years ago

Thank you, @alexforencich. This is puzzling because the difference between these two is supposed to be in memory usage (16 MiB vs. 1 GiB):

scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED

Further, it explicitly knows it "FAILED" - it does not simply return a miscomputed value. This is why I thought of memory allocation error as the most likely cause.

Even further, the value it returned for the 16 MiB case is the correct one.

If not total RAM, then maybe e.g. a low RLIMIT_AS?

solardiz commented 2 years ago

Oh, here's another idea - maybe something weird goes on with allocation of huge pages. We have this in lib/alg-yescrypt-platform.c:

#define HUGEPAGE_THRESHOLD              (32 * 1024 * 1024)

Obviously, one of these values is below and the other above this threshold. Can you try playing with this threshold to see if it makes a difference - e.g., replace the 32 with 1500, so that neither test uses huge pages?

alexforencich commented 2 years ago

Aha, possibly something strange is going on with hugepages. On the machines in question, hugepages are configured like so:

default_hugepagesz=1G hugepagesz=1G hugepages=16

So if the tests are trying to allocate too many huge pages, then something will fail.

I'll make that change and see what happens when re-running the tests.

solardiz commented 2 years ago

The code assumes that there are 2M-sized huge pages, and maybe you only have 1G-sized instead? The code aligns the size for 2M, not for 1G, so the allocation will likely fail for that reason. However, it should then retry without huge pages at all. So it's still puzzling why it fails visibly.

alexforencich commented 2 years ago

Tests pass with that change. Definitely looks like there is something wrong with how hugepages are handled here; it looks like the code is expecting hugepages to be 2 MB, but on those machines hugepages are 1 GB.

alexforencich commented 2 years ago

On the problematic machine:

$ cat /proc/meminfo | grep -i huge
AnonHugePages:     18432 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:      16
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
Hugetlb:        16777216 kB

On a machine that works:

AnonHugePages:     73728 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
solardiz commented 2 years ago

As I understand, Zen 2 (and thus EPYC) does support 2 MB pages as well - so it's a matter of kernel configurations.

Regardless, I don't see why the expected fallback to non-huge pages apparently also failed, and why some of the gensalt tests failed - these should be unrelated, but you say adjusting the threshold fixed them as well?

alexforencich commented 2 years ago

After making that change, all of the tests passed. Yes, the machines probably support 2 MB hugepages, but as the machines are used for networking applications with DPDK, the hugepage size is set to 1 GB. But an encryption library should not be dependent on the kernel configuration in this way.

Also, I did a bit of poking around, and the alg-yescrypt test binary reports a segfault at the end of the test. I'm assuming this is probably a result of attempting to free a buffer that wasn't actually allocated in the first place or something along those lines, but I'll see if I can get a stack trace.

solardiz commented 2 years ago

why some of the gensalt tests failed

Oh, I think I understand - they were followed by crypt using the generated salt, and that failed. OK then.

So we just need to figure out what happened with the fallback, which is here:

    base = mmap(NULL, new_size, PROT_READ | PROT_WRITE, flags, -1, 0);
    if (base != MAP_FAILED) {
        base_size = new_size;
    } else if (flags & MAP_HUGETLB) {
        flags &= ~MAP_HUGETLB;
        base = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
    }

The expectation is that we'd get MAP_FAILED the first time, and then retry without MAP_HUGETLB and it'd succeed.

alexforencich commented 2 years ago

What file is that in? I can add some prints and run it again.

solardiz commented 2 years ago

This is in lib/alg-yescrypt-platform.c. Maybe gcc 11 somehow "optimizes" this. We have a comparison between pointers to different types in there. Maybe try casting (void *)base in that if?

alexforencich commented 2 years ago

Segfault backtrace:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7ef54bd in __strlen_avx2 () from /usr/lib/libc.so.6
(gdb) where
#0  0x00007ffff7ef54bd in __strlen_avx2 () from /usr/lib/libc.so.6
#1  0x00007ffff7e120c3 in strdup () from /usr/lib/libc.so.6
#2  0x00005555555555f9 in main ()
(gdb) list
solardiz commented 2 years ago

The segfault is because the test (not the code being tested) is careless - it does not check for NULL return from yescrypt (which indicates it has failed, and we already almost know why), passing the pointer into strdup.

                uint8_t *hash = yescrypt(
                    (const uint8_t *)"pleaseletmein", setting);
                printf("Plaintext: '%s'\n", (char *)hash);
                hash = (uint8_t *)strdup((char *)hash);

We could fix this or not. The real important fix should be to the actual code being tested, to make the fallback work.

alexforencich commented 2 years ago

So I instrumented the code like so:

base = mmap(NULL, new_size, PROT_READ | PROT_WRITE, flags, -1, 0);
printf("mmap res 1: %p\n", base);
if (base != MAP_FAILED) {
    printf("mmap succeeded\n");
    base_size = new_size;
} else if (flags & MAP_HUGETLB) {
    printf("mmap failed, try without hugepages\n");
    flags &= ~MAP_HUGETLB;
    base = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
    printf("mmap res 2: %p\n", base);
}

And I am getting:

scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) =mmap res 1: 0x7f38c0000000
mmap succeeded
 FAILED

So it looks like mmap succeeded, but it failed somewhere else?

solardiz commented 2 years ago

Looks so!

Meanwhile, I've just tested the fallback by making the first mmap fail - the fallback worked. This is with gcc 11.2.1.

solardiz commented 2 years ago

@alexforencich Are you sure you reverted the threshold adjustment first - that is, that MAP_HUGETLB was set in the first mmap call in your test above? Maybe add printing of that flag as well.

alexforencich commented 2 years ago

I reverted the adjustment, and if you look at the pointer LSBs, it definitely looks like it allocated a 1GB huge page:

scrypt("", "", 16, 1, 1) =mmap res 1: 0x7f3961a3f000
mmap succeeded
 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
scrypt("password", "NaCl", 1024, 8, 16) =mmap res 1: 0x7f39616b5000
mmap succeeded
 fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) =mmap res 1: 0x7f39607b9000
mmap succeeded
 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) =mmap res 1: 0x7f38c0000000
mmap succeeded
 FAILED
yescrypt("", "", 0, 16, 1, 1, 0, 0) =mmap res 1: 0x7f3961a3f000
mmap succeeded
 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
yescrypt("", "", 0, 16, 1, 1, 0, 0) =mmap res 1: 0x7f3961a3f000
mmap succeeded
solardiz commented 2 years ago

Oh, right. I assume you can also reproduce the same by:

git clone https://github.com/openwall/yescrypt
cd yescrypt
make check
alexforencich commented 2 years ago

yep, that one seems to report the same failure:

scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
solardiz commented 2 years ago

I cannot reproduce the problem easily, so would appreciate it if you figure it out on this smaller codebase. It should be rather straightforward to see where the code detects it's failing for you.

alexforencich commented 2 years ago

Would remote access to one of the machines in question be helpful?

alexforencich commented 2 years ago

Alright, here's something strange: I just did some testing of that yescrypt code on two other machines (with Xeon CPUs), otherwise identical to each other except for the hugepages configuration (one is default, one has 16x 1G hugepages), and I'm not seeing the same failure there. However, the test case fails somewhere else. But these machines are running Ubuntu 18.04 instead of Arch.

solardiz commented 2 years ago

Here's my current best guess: the failure is on munmap().

Maybe this is related to the size we're unmapping not exactly matching the size the kernel actually maps. Specifically, we call mmap() with a size of 1 GiB + 2 MiB (in that test, we need a tiny bit more than 1 GiB, and this gets rounded up to the next multiple of 2 MiB due to our setting of HUGEPAGE_SIZE). We then also call munmap() with that same size, but maybe the kernel finally insists on us using multiples of its actual huge page size at this point (and maybe it actually allocated two such pages, although ideally we'd have had one 1 GiB page followed by one 4 KiB page).

@alexforencich You should be able to easily confirm this with strace. You'd also see the sizes used in those calls, and you can try increasing HUGEPAGE_SIZE in yescrypt-platform.c to 1 GiB.

If confirmed, a question would be whether this is a kernel bug. Arguably, if it's going to disallow munmap() at that size, it should have failed on the mmap() - IIRC, that's the behavior we saw with 2 MiB pages when we were not rounding up the size.

Another question would be what to do about this, if at all. Share a reduced test case with Linux kernel maintainers?

I don't currently have a good idea for an efficient and safe workaround we could include. Setting our HUGEPAGE_SIZE to 1 GiB would probably work, but would be very wasteful on systems that use 2 MiB. Retrying munmap() with rounding up to 1 GiB is unsafe - we'd risk unmapping something unrelated that just happens to be right after our mapping.

Would remote access to one of the machines in question be helpful?

If my guesses above are all correct, then I don't currently see what else I'd test - short of debugging kernel internals. Otherwise, yes, I'd see where else it fails - although I don't currently see any other realistic possibility.

I'm not seeing the same failure there. However, the test case fails somewhere else.

Can you show where it fails on Intel? I assume also only on the machine with 1G pages?

alexforencich commented 2 years ago

Running the alg-yescrypt test with strace results in:

mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0) = 0x7f11c0000000
write(1, "scrypt(\"pleaseletmein\", \"SodiumC"..., 85scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) =mmap res 1: 0x7f11c0000000
) = 85
write(1, "mmap succeeded\n", 15mmap succeeded
)        = 15
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
--- SIGWINCH {si_signo=SIGWINCH, si_code=SI_KERNEL} ---
munmap(0x7f11c0000000, 1075838976)      = -1 EINVAL (Invalid argument)
write(1, " FAILED\n", 8 FAILED
)                = 8

I have not tried changing the hugepage size yet. But yes, this certainly could be a kernel bug, and that would explain why I'm not seeing the same issue on the two Intel servers, despite one of them being configured for a huge page size of 1 GB, since they're running a significantly older kernel.

The two Intel machines seem to be returning the same test results:

$ make check
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO yescrypt-opt.c
yescrypt-opt.c:42:2: warning: #warning "Note: AVX and XOP are not enabled, which is great for YESCRYPT_RW, but they would substantially improve performance at classic scrypt and YESCRYPT_WORM" [-Wcpp]
 #warning "Note: AVX and XOP are not enabled, which is great for YESCRYPT_RW, but they would substantially improve performance at classic scrypt and YESCRYPT_WORM"
  ^~~~~~~
yescrypt-opt.c:511:2: warning: #warning "Note: using x86-64 inline assembly for YESCRYPT_RW.  That's great." [-Wcpp]
 #warning "Note: using x86-64 inline assembly for YESCRYPT_RW.  That's great."
  ^~~~~~~
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO yescrypt-common.c
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO sha256.c
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO insecure_memzero.c
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO tests.c
gcc -s -lrt -fopenmp yescrypt-opt.o yescrypt-common.o sha256.o insecure_memzero.o tests.o -o tests
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO -DTEST phc.c -o phc-test.o
gcc -s -lrt -fopenmp yescrypt-opt.o yescrypt-common.o sha256.o insecure_memzero.o phc-test.o -o phc-test
Running main tests
/bin/sh: 1: time: not found
--- TESTS-OK    2022-02-19 14:24:46.084345030 -0800
+++ TESTS-OUT   2022-02-20 11:23:20.810817960 -0800
@@ -1,80 +0,0 @@
-scrypt("", "", 16, 1, 1) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
-scrypt("password", "NaCl", 1024, 8, 16) = fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
-scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
-scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = 21 01 cb 9b 6a 51 1a ae ad db be 09 cf 70 f8 81 ec 56 8d 57 4a 2f fd 4d ab e5 ee 98 20 ad aa 47 8e 56 fd 8f 4b a5 d0 9f fa 1c 6d 92 7c 40 f4 c3 37 30 40 49 e8 a9 52 fb cb f4 5c 6f a7 7a 41 a4
-yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
-yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20
-yescrypt("", "", 0, 4, 1, 1, 0, 0) = ef ad 0c 23 31 4c b5 72 bc 3c fb 15 43 da 42 f8 a8 b0 73 00 4c 86 6b 64 ab 50 55 a4 f0 9f a5 f5 71 14 2e bf e7 e0 5a 3b 92 c4 32 f3 1d ea 95 ad 5f 9c 85 4b 64 56 46 2f 4b d0 f7 32 b7 cd c5 49
-yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f 7f 1a e8 b4 df ed a5 1f 8b 6d 56 f3 08 1b e1 a7 c0 83 3b a2 71 9a 36 ab 02 88 5d ae 36 55 7d 34 26 86 b1 7b a7 5f 2c 21 77 92 de 09 70 ab 1d 07 a9 c7 50 93 6d 31 42 6f
-yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f
-yescrypt("", "", 1, 4, 1, 1, 1, 0) = 4b aa 8c d8 60 8b a9 1f 3e 34 39 d9 ec 4f ae 8f 9f c0 92 d9 ca 22 b7 37 7e 31 ae 5b 9a d7 87 7c 11 68 69 11 62 dd 0e 5e f0 49 e5 70 65 0c be d4 38 4a d6 05 34 fb 0c be d1 9f f3 f0 33 c9 4b 0c
-yescrypt("", "", 1, 4, 1, 1, 2, 0) = e6 e8 bb a0 9b 64 12 ff b0 b3 cc 35 e3 7d 0b 78 2a 47 fb aa dc 57 a0 76 d7 c6 cc 2e 70 91 9a 1b 8d 47 38 c4 f8 33 55 69 07 42 d9 be d7 1c 3b 8f b0 d7 eb 08 6a b1 34 c5 e5 57 07 c2 c1 3c 75 ef
-yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99 7d 70 ae 0a 64 bf 0a 4d 96 c1 73 ab f8 82 79 c1 a9 4a d9 bd f1 68 ed fb bd 90 f6 6e d5 c8 0d
-yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99
-yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08
-yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1
-yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac
-yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76 eb 24 1d f8 11 9a 9a 12 2a e3 69 20 bc c7 f4 14 b9 c0 d5 8f 45 00 80 60 da de 46 b0 c8 09 22 bd cc 16 a3 ab 5d 20 1d 4c 61 40 c6 71 be 1f 75 27 2c a9 04 73 9d 5a d1 ff 67 2b 0c 21
-yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76
-yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68 ca 36 2c 55 57 cc 04 b6 81 1e 2e 73 08 41 f5 26 d8 f4 f7 ac fb fa 9e 06 fe 1f 38 3a 71 15 5e
-yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68
-yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4
-yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35
-yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23
-yescrypt("p", "s", 182, 16, 8, 1, 10, 0) = e1 f9 81 73 3a 94 05 2f cd 7a cb 14 05 df 0b bd e8 e4 99 b6 a1 33 1b 77 59 09 b4 8c 2f 51 6c 40 dc c8 30 16 35 b7 23 7b
-yescrypt("p", "s", 1, 16, 8, 1, 10, 0) = 9e 7a 40 97 64 42 84 cf 3b 73 b6 04 50 ff 23 0c dc b6 b1 b1 9b 15 09 ee b4 82 f6 96 c4 f1 c7 05 c0 0f 74 02 16 18 3a 12
-yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99 c3 d6 69 30 fb 20 23 bb
-yescrypt("p", "s", 1, 16, 8, 1, 0, 0) = 9d d6 36 c2 d0 bb 92 34 52 86 ef da f8 a6 8c fc 1b 4f fd c4 b1 ad ac cc 7d 86 4b 9a 67 87 b8 5d 6a e0 f5 28 0d a8 88 9f
-yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99
-yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91
-'$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.'
-Plaintext: '$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.$HboGM6qPrsK.StKYGt6KErmUYtioHreJd98oIugoNB6'
-Encrypted: '$y$jD5.7$BkbiDbyWZnzlLWOAcru671$zLAHafRUyp9n9XZWnltUbj3ULWUtMN4fteTltjWkARC'
-Plaintext: '$y$jC4$LdJMENpBABJJ3hIHjB1B$jVg4HoqqpbmQv/NCpin.QCMagJ8o4QX7lXdzvVV0xFC'
-Encrypted: '$y$jC4$qiyh2SQgE5vrF3ORvFho$HurI7MuukXHz..TpxrwKuakji/j9VKDh2WVUK4DIsq5'
-Plaintext: '$y$/B3.6$LdJMENpBABJJ3hIHjB1$h8sE4hJo.BsdlfJr0.d8bNJNPZymH7Y3kLj4aY1Rfc8'
-Encrypted: '$y$/B3.6$YiN5s/dKpjNrdKm9ND0$lBNMoUaAsw.JR0zMq9IBKIi/VPxj7lD7Sg64nB5LFw2'
-Plaintext: '$y$/A2$LdJMENpBABJJ3hIHj/$5IEld1eWdmh5lylrqHLF5dvA3ISpimEM9J1Dd05n/.3'
-Encrypted: '$y$/A2$TqjvmGXoqnsNU/8Y40$ZB43..7UnMt6ySU7XbpPyvyahahHSkJJgztkLWp6/IC'
-Plaintext: '$y$j91.5$LdJMENpBABJJ3hIH$ebKnn23URD5vyLgF9cP2EvVosrUXf7UErGRV0KmC6e6'
-Encrypted: '$y$j91.5$bICydADAaInC9UR/$rpsDDkULkr1caCVYV9PNAsiZxijoQ2/gDcbonlowRi7'
-Plaintext: '$y$j80$LdJMENpBABJJ3h2$ysXVVJwuaVlI1BWoEKt/Bz3WNDDmdOWz/8KTQaHL1cC'
-Encrypted: '$y$j80$fmxyeGYOC34lh19$mm9FRBs0iHLTHfSNznm1kJVchXN4PaS8hoNI6TTAlB7'
-Plaintext: '$y$/7/.4$LdJMENpBABJJ3/$lXHleh7bIZMGNtJVxGVrsIWkEIXfBedlfPui/PITflC'
-Encrypted: '$y$/7/.4$EuTbL3Wtm3khW0$1jvKQzIcAqYnYxfb4TEs.FeAZ7rLDl5vNQEkPPcj2KC'
-Plaintext: '$y$/6.$LdJMENpBABJJ$zQITmYSih5.CTY47x0IuE4wl.b3HzYGKKCSggakaQ22'
-Encrypted: '$y$/6.$x0G/jIr053ui$4O.aVGTPptkjx6eXrW8fdvqcPEV28w7a1PSos6CXV31'
-Plaintext: '$y$j5..3$LdJMENpBAB3$xi27PTUNd8NsChHeLOz85JFnUOyibRHkWzprowRlR5/'
-Encrypted: '$y$j5..3$/nwg3UXJWp/$5jcvDgeotKpaG9IeSJx0fJNSz33JjTYYD4Kwao3Eki5'
-Plaintext: '$y$j4/$LdJMENpBA/$tHlkpTQ8V/eEnTVau1uW36T97LIXlfPrEzdeV5SE5K7'
-Encrypted: '$y$j4/$yoneNBwae0$uPBnH0yXBCOM5v5BU9qlvUUtUr3QD5btS0upc6sdvf4'
-Plaintext: '$y$/3..2$LdJMENpB$tNczXFuNUd3HMqypStCRsEaL4e4KF7ZYLBe8Hbeg0B7'
-Encrypted: '$y$/3..2$VD20uHT3$AV5WWaN6bEKRvZlCuurj.mnHMmZmJ9ExQ9HjiReCDwC'
-Plaintext: '$y$/2/$LdJMEN3$RRorHhfsw1/P/WR6Aurg4U72e9Q7qt9vFPURdyfiqK8'
-Encrypted: '$y$/2/$BYujKJA$fsMwVvFm8r1caFQP.mem3OUuMYBCDGj9CEoDfSwFDLB'
-Plaintext: '$y$j2..1$LdJME/$iLEt6kuTwHch6XdCxtTHfsQzYwWFmpUwgl6Ax8RH4d1'
-Encrypted: '$y$j2..1$.mZga/$X6GFMkoYPxFapo.3H4LllEjltFapONQcKUOdEd9oPa/'
-Plaintext: '$y$j0/$LdJM$k7BXzSDuoGHW56SY3HxROCiA0gWRscZe2aA0q5oHPM0'
-Encrypted: '$y$j0/$SkNZ$DQ06H0br45bpE7lGgCD9gOxTMP9SsO6Mt1T9lo5PHz1'
-Plaintext: '$y$//..0$Ld3$6BJXezMFxaMiO5wsuoEmztvtCs/79085dZO56ADlV5B'
-Encrypted: '$y$//..0$lM1$60gjeUIW/3QidfN6zU9NqB09Ni1NBMfj2VaSZMjDd18'
-Plaintext: '$y$///$L/$Rrrkp6OVljrIk0kcwkCDhAiHJiSthh3cKeIGHUW7Z0C'
-Encrypted: '$y$///$q.$/.tR4GqigxciLYGoB8fmzudWQR7IzSu9s3dR8wp3VsD'
-Plaintext: '$y$j1../$LdJMENpBABJJ3hIHjB1Bi.$L8OQFc8mxJPd7CpUFgkS7KqJM2I9jGXu3BdqX2D.647'
-Encrypted: '$y$j1../$BkbiDbyWZnzlLWOAcru671$iicGI2gNZyhimPVgz2VoKrJAB9fWykBN.3Mh0AwEy29'
-Plaintext: '$y$j//$LdJMENpBABJJ3hIHjB1B$U8a2MaK.yesqWySK8Owk6PWeWmp/XuagMbpP45q1/q1'
-Encrypted: '$y$j//$qiyh2SQgE5vrF3ORvFho$5dD9ick8ugystfp8wa3xbV7ASDux0dpoOh0QJxFuXH5'
-'$7$C6..../....SodiumChloride$kBGj9fHznVYFQMEn/qDCfrDevf9YDtcDdKvEqHJLV8D'
-'$7$06..../....SodiumChloride$ENlyo6fGw4PCcDBOFepfSZjFUnVatHzCcW55.ZGz3B0'
-r=8 N=2^11 NROM=2^18
-Will use 262144.00 KiB ROM
-         2048.00 KiB RAM
-Initializing ROM ... DONE (696ebab2)
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.'
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
-Initializing ROM in preallocated memory ... DONE (696ebab2)
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$K3wFVK9/t3QsjCk/oK2s8dKzzZ4m7QTP8Ms5uywhWv8'
-'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$/OnmIkP0UK5OxyxD0Af/V1oL0zWvTLAUWg3Nr0bsFEB'
-'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$DskEGULspNduIZVFK5SOK8enlXnSs/vkuXFdi0wkQ1.'
-'$y$j/.5I$LdJMENpBABJJ3hIHjB1Bi.$NqCMKxN9Y9Uw821.72ScGDMpyk7U7V51qnHSRPapzW8'
FAILED
Running PHC tests
/bin/sh: 3: time: not found
PHC-TEST-OUT: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
Makefile:43: recipe for target 'check' failed
make: *** [check] Error 1
$ cat /proc/meminfo | grep -i huge
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:      16
HugePages_Free:       16
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
$ uname -a
Linux reactor8 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$

Seems like an unrelated test is failing for some reason; it is reported as failing on both machines even though only one as hugepages set to 1G. Additionally, there seems to be an issue with one of the test scripts finding 'time'.

This is what I get on the AMD machine:

$ make check
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO yescrypt-opt.c
yescrypt-opt.c:42:2: warning: #warning "Note: AVX and XOP are not enabled, which is great for YESCRYPT_RW, but they would substantially improve performance at classic scrypt and YESCRYPT_WORM" [-Wcpp]
   42 | #warning "Note: AVX and XOP are not enabled, which is great for YESCRYPT_RW, but they would substantially improve performance at classic scrypt and YESCRYPT_WORM"
      |  ^~~~~~~
yescrypt-opt.c:511:2: warning: #warning "Note: using x86-64 inline assembly for YESCRYPT_RW.  That's great." [-Wcpp]
  511 | #warning "Note: using x86-64 inline assembly for YESCRYPT_RW.  That's great."
      |  ^~~~~~~
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO yescrypt-common.c
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO sha256.c
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO insecure_memzero.c
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO tests.c
gcc -s -lrt -fopenmp yescrypt-opt.o yescrypt-common.o sha256.o insecure_memzero.o tests.o -o tests
gcc -c -Wall -O2 -fomit-frame-pointer -fopenmp -DSKIP_MEMZERO -DTEST phc.c -o phc-test.o
gcc -s -lrt -fopenmp yescrypt-opt.o yescrypt-common.o sha256.o insecure_memzero.o phc-test.o -o phc-test
Running main tests
scrypt("", "", 16, 1, 1) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
scrypt("password", "NaCl", 1024, 8, 16) = fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20
yescrypt("", "", 0, 4, 1, 1, 0, 0) = ef ad 0c 23 31 4c b5 72 bc 3c fb 15 43 da 42 f8 a8 b0 73 00 4c 86 6b 64 ab 50 55 a4 f0 9f a5 f5 71 14 2e bf e7 e0 5a 3b 92 c4 32 f3 1d ea 95 ad 5f 9c 85 4b 64 56 46 2f 4b d0 f7 32 b7 cd c5 49
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f 7f 1a e8 b4 df ed a5 1f 8b 6d 56 f3 08 1b e1 a7 c0 83 3b a2 71 9a 36 ab 02 88 5d ae 36 55 7d 34 26 86 b1 7b a7 5f 2c 21 77 92 de 09 70 ab 1d 07 a9 c7 50 93 6d 31 42 6f
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f
yescrypt("", "", 1, 4, 1, 1, 1, 0) = 4b aa 8c d8 60 8b a9 1f 3e 34 39 d9 ec 4f ae 8f 9f c0 92 d9 ca 22 b7 37 7e 31 ae 5b 9a d7 87 7c 11 68 69 11 62 dd 0e 5e f0 49 e5 70 65 0c be d4 38 4a d6 05 34 fb 0c be d1 9f f3 f0 33 c9 4b 0c
yescrypt("", "", 1, 4, 1, 1, 2, 0) = e6 e8 bb a0 9b 64 12 ff b0 b3 cc 35 e3 7d 0b 78 2a 47 fb aa dc 57 a0 76 d7 c6 cc 2e 70 91 9a 1b 8d 47 38 c4 f8 33 55 69 07 42 d9 be d7 1c 3b 8f b0 d7 eb 08 6a b1 34 c5 e5 57 07 c2 c1 3c 75 ef
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99 7d 70 ae 0a 64 bf 0a 4d 96 c1 73 ab f8 82 79 c1 a9 4a d9 bd f1 68 ed fb bd 90 f6 6e d5 c8 0d
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76 eb 24 1d f8 11 9a 9a 12 2a e3 69 20 bc c7 f4 14 b9 c0 d5 8f 45 00 80 60 da de 46 b0 c8 09 22 bd cc 16 a3 ab 5d 20 1d 4c 61 40 c6 71 be 1f 75 27 2c a9 04 73 9d 5a d1 ff 67 2b 0c 21
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68 ca 36 2c 55 57 cc 04 b6 81 1e 2e 73 08 41 f5 26 d8 f4 f7 ac fb fa 9e 06 fe 1f 38 3a 71 15 5e
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23
yescrypt("p", "s", 182, 16, 8, 1, 10, 0) = e1 f9 81 73 3a 94 05 2f cd 7a cb 14 05 df 0b bd e8 e4 99 b6 a1 33 1b 77 59 09 b4 8c 2f 51 6c 40 dc c8 30 16 35 b7 23 7b
yescrypt("p", "s", 1, 16, 8, 1, 10, 0) = 9e 7a 40 97 64 42 84 cf 3b 73 b6 04 50 ff 23 0c dc b6 b1 b1 9b 15 09 ee b4 82 f6 96 c4 f1 c7 05 c0 0f 74 02 16 18 3a 12
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99 c3 d6 69 30 fb 20 23 bb
yescrypt("p", "s", 1, 16, 8, 1, 0, 0) = 9d d6 36 c2 d0 bb 92 34 52 86 ef da f8 a6 8c fc 1b 4f fd c4 b1 ad ac cc 7d 86 4b 9a 67 87 b8 5d 6a e0 f5 28 0d a8 88 9f
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91
'$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.'
Plaintext: '(null)'

real    0m11.922s
user    0m9.928s
sys 0m3.375s
--- TESTS-OK    2022-02-19 14:01:17.545889233 -0800
+++ TESTS-OUT   2022-02-20 11:28:41.358912391 -0800
@@ -4 +4 @@
-scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = 21 01 cb 9b 6a 51 1a ae ad db be 09 cf 70 f8 81 ec 56 8d 57 4a 2f fd 4d ab e5 ee 98 20 ad aa 47 8e 56 fd 8f 4b a5 d0 9f fa 1c 6d 92 7c 40 f4 c3 37 30 40 49 e8 a9 52 fb cb f4 5c 6f a7 7a 41 a4
+scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
@@ -31,50 +31 @@
-Plaintext: '$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.$HboGM6qPrsK.StKYGt6KErmUYtioHreJd98oIugoNB6'
-Encrypted: '$y$jD5.7$BkbiDbyWZnzlLWOAcru671$zLAHafRUyp9n9XZWnltUbj3ULWUtMN4fteTltjWkARC'
-Plaintext: '$y$jC4$LdJMENpBABJJ3hIHjB1B$jVg4HoqqpbmQv/NCpin.QCMagJ8o4QX7lXdzvVV0xFC'
-Encrypted: '$y$jC4$qiyh2SQgE5vrF3ORvFho$HurI7MuukXHz..TpxrwKuakji/j9VKDh2WVUK4DIsq5'
-Plaintext: '$y$/B3.6$LdJMENpBABJJ3hIHjB1$h8sE4hJo.BsdlfJr0.d8bNJNPZymH7Y3kLj4aY1Rfc8'
-Encrypted: '$y$/B3.6$YiN5s/dKpjNrdKm9ND0$lBNMoUaAsw.JR0zMq9IBKIi/VPxj7lD7Sg64nB5LFw2'
-Plaintext: '$y$/A2$LdJMENpBABJJ3hIHj/$5IEld1eWdmh5lylrqHLF5dvA3ISpimEM9J1Dd05n/.3'
-Encrypted: '$y$/A2$TqjvmGXoqnsNU/8Y40$ZB43..7UnMt6ySU7XbpPyvyahahHSkJJgztkLWp6/IC'
-Plaintext: '$y$j91.5$LdJMENpBABJJ3hIH$ebKnn23URD5vyLgF9cP2EvVosrUXf7UErGRV0KmC6e6'
-Encrypted: '$y$j91.5$bICydADAaInC9UR/$rpsDDkULkr1caCVYV9PNAsiZxijoQ2/gDcbonlowRi7'
-Plaintext: '$y$j80$LdJMENpBABJJ3h2$ysXVVJwuaVlI1BWoEKt/Bz3WNDDmdOWz/8KTQaHL1cC'
-Encrypted: '$y$j80$fmxyeGYOC34lh19$mm9FRBs0iHLTHfSNznm1kJVchXN4PaS8hoNI6TTAlB7'
-Plaintext: '$y$/7/.4$LdJMENpBABJJ3/$lXHleh7bIZMGNtJVxGVrsIWkEIXfBedlfPui/PITflC'
-Encrypted: '$y$/7/.4$EuTbL3Wtm3khW0$1jvKQzIcAqYnYxfb4TEs.FeAZ7rLDl5vNQEkPPcj2KC'
-Plaintext: '$y$/6.$LdJMENpBABJJ$zQITmYSih5.CTY47x0IuE4wl.b3HzYGKKCSggakaQ22'
-Encrypted: '$y$/6.$x0G/jIr053ui$4O.aVGTPptkjx6eXrW8fdvqcPEV28w7a1PSos6CXV31'
-Plaintext: '$y$j5..3$LdJMENpBAB3$xi27PTUNd8NsChHeLOz85JFnUOyibRHkWzprowRlR5/'
-Encrypted: '$y$j5..3$/nwg3UXJWp/$5jcvDgeotKpaG9IeSJx0fJNSz33JjTYYD4Kwao3Eki5'
-Plaintext: '$y$j4/$LdJMENpBA/$tHlkpTQ8V/eEnTVau1uW36T97LIXlfPrEzdeV5SE5K7'
-Encrypted: '$y$j4/$yoneNBwae0$uPBnH0yXBCOM5v5BU9qlvUUtUr3QD5btS0upc6sdvf4'
-Plaintext: '$y$/3..2$LdJMENpB$tNczXFuNUd3HMqypStCRsEaL4e4KF7ZYLBe8Hbeg0B7'
-Encrypted: '$y$/3..2$VD20uHT3$AV5WWaN6bEKRvZlCuurj.mnHMmZmJ9ExQ9HjiReCDwC'
-Plaintext: '$y$/2/$LdJMEN3$RRorHhfsw1/P/WR6Aurg4U72e9Q7qt9vFPURdyfiqK8'
-Encrypted: '$y$/2/$BYujKJA$fsMwVvFm8r1caFQP.mem3OUuMYBCDGj9CEoDfSwFDLB'
-Plaintext: '$y$j2..1$LdJME/$iLEt6kuTwHch6XdCxtTHfsQzYwWFmpUwgl6Ax8RH4d1'
-Encrypted: '$y$j2..1$.mZga/$X6GFMkoYPxFapo.3H4LllEjltFapONQcKUOdEd9oPa/'
-Plaintext: '$y$j0/$LdJM$k7BXzSDuoGHW56SY3HxROCiA0gWRscZe2aA0q5oHPM0'
-Encrypted: '$y$j0/$SkNZ$DQ06H0br45bpE7lGgCD9gOxTMP9SsO6Mt1T9lo5PHz1'
-Plaintext: '$y$//..0$Ld3$6BJXezMFxaMiO5wsuoEmztvtCs/79085dZO56ADlV5B'
-Encrypted: '$y$//..0$lM1$60gjeUIW/3QidfN6zU9NqB09Ni1NBMfj2VaSZMjDd18'
-Plaintext: '$y$///$L/$Rrrkp6OVljrIk0kcwkCDhAiHJiSthh3cKeIGHUW7Z0C'
-Encrypted: '$y$///$q.$/.tR4GqigxciLYGoB8fmzudWQR7IzSu9s3dR8wp3VsD'
-Plaintext: '$y$j1../$LdJMENpBABJJ3hIHjB1Bi.$L8OQFc8mxJPd7CpUFgkS7KqJM2I9jGXu3BdqX2D.647'
-Encrypted: '$y$j1../$BkbiDbyWZnzlLWOAcru671$iicGI2gNZyhimPVgz2VoKrJAB9fWykBN.3Mh0AwEy29'
-Plaintext: '$y$j//$LdJMENpBABJJ3hIHjB1B$U8a2MaK.yesqWySK8Owk6PWeWmp/XuagMbpP45q1/q1'
-Encrypted: '$y$j//$qiyh2SQgE5vrF3ORvFho$5dD9ick8ugystfp8wa3xbV7ASDux0dpoOh0QJxFuXH5'
-'$7$C6..../....SodiumChloride$kBGj9fHznVYFQMEn/qDCfrDevf9YDtcDdKvEqHJLV8D'
-'$7$06..../....SodiumChloride$ENlyo6fGw4PCcDBOFepfSZjFUnVatHzCcW55.ZGz3B0'
-r=8 N=2^11 NROM=2^18
-Will use 262144.00 KiB ROM
-         2048.00 KiB RAM
-Initializing ROM ... DONE (696ebab2)
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.'
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
-Initializing ROM in preallocated memory ... DONE (696ebab2)
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$K3wFVK9/t3QsjCk/oK2s8dKzzZ4m7QTP8Ms5uywhWv8'
-'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$/OnmIkP0UK5OxyxD0Af/V1oL0zWvTLAUWg3Nr0bsFEB'
-'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$DskEGULspNduIZVFK5SOK8enlXnSs/vkuXFdi0wkQ1.'
-'$y$j/.5I$LdJMENpBABJJ3hIHjB1Bi.$NqCMKxN9Y9Uw821.72ScGDMpyk7U7V51qnHSRPapzW8'
+Plaintext: '(null)'
FAILED
Running PHC tests
m_cost=0 (8 KiB), t_cost=0
6450 c/s real, 12900 c/s virtual (258 hashes in 0.04 seconds)
m_cost=7 (1024 KiB), t_cost=0
548 c/s real, 537 c/s virtual (258 hashes in 0.47 seconds)
m_cost=8 (2048 KiB), t_cost=0
283 c/s real, 283 c/s virtual (258 hashes in 0.91 seconds)
m_cost=8 (2048 KiB), t_cost=1
238 c/s real, 241 c/s virtual (258 hashes in 1.08 seconds)
m_cost=8 (2048 KiB), t_cost=2
204 c/s real, 204 c/s virtual (258 hashes in 1.26 seconds)
m_cost=8 (2048 KiB), t_cost=3
145 c/s real, 146 c/s virtual (258 hashes in 1.77 seconds)
m_cost=11 (16384 KiB), t_cost=0
33 c/s real, 33 c/s virtual (258 hashes in 7.72 seconds)

real    0m13.253s
user    0m10.800s
sys 0m2.429s
PHC-TEST-OUT: OK

And strace shows:

write(1, "scrypt(\"pleaseletmein\", \"SodiumC"..., 249scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
) = 249
mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0) = 0x7f6280000000
futex(0x55926357e854, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x55926357e854, FUTEX_WAKE_PRIVATE, 2147483647) = 0
munmap(0x7f6280000000, 1075838976)      = -1 EINVAL (Invalid argument)
write(1, "scrypt(\"pleaseletmein\", \"SodiumC"..., 66scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
) = 66
solardiz commented 2 years ago

Additionally, there seems to be an issue with one of the test scripts finding 'time'.

I think that's actually why it's failing - can you try installing time?

alexforencich commented 2 years ago

That's what's odd, it seems to be installed:

$ time

real    0m0.000s
user    0m0.000s
sys 0m0.000s
alexforencich commented 2 years ago

Hmm, there is something strange going on here as well. If I run the PHC test manually, it works:

$ ./phc-test > PHC-TEST-OUT
m_cost=0 (8 KiB), t_cost=0
6450 c/s real, 8600 c/s virtual (258 hashes in 0.04 seconds)
m_cost=7 (1024 KiB), t_cost=0
600 c/s real, 600 c/s virtual (258 hashes in 0.43 seconds)
m_cost=8 (2048 KiB), t_cost=0
379 c/s real, 379 c/s virtual (258 hashes in 0.68 seconds)
m_cost=8 (2048 KiB), t_cost=1
344 c/s real, 339 c/s virtual (258 hashes in 0.75 seconds)
m_cost=8 (2048 KiB), t_cost=2
300 c/s real, 303 c/s virtual (258 hashes in 0.86 seconds)
m_cost=8 (2048 KiB), t_cost=3
220 c/s real, 220 c/s virtual (258 hashes in 1.17 seconds)
m_cost=11 (16384 KiB), t_cost=0
46 c/s real, 46 c/s virtual (258 hashes in 5.53 seconds)
$ sha256sum -c PHC-TEST-OK-SHA256
PHC-TEST-OUT: OK

But, if I run the other set of tests manually, it fails on the 1G hugepage machine, but fails on the other one.

First machine, with 1G huge pages:

$ time ./tests | tee TESTS-OUT
scrypt("", "", 16, 1, 1) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
scrypt("password", "NaCl", 1024, 8, 16) = fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20
yescrypt("", "", 0, 4, 1, 1, 0, 0) = ef ad 0c 23 31 4c b5 72 bc 3c fb 15 43 da 42 f8 a8 b0 73 00 4c 86 6b 64 ab 50 55 a4 f0 9f a5 f5 71 14 2e bf e7 e0 5a 3b 92 c4 32 f3 1d ea 95 ad 5f 9c 85 4b 64 56 46 2f 4b d0 f7 32 b7 cd c5 49
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f 7f 1a e8 b4 df ed a5 1f 8b 6d 56 f3 08 1b e1 a7 c0 83 3b a2 71 9a 36 ab 02 88 5d ae 36 55 7d 34 26 86 b1 7b a7 5f 2c 21 77 92 de 09 70 ab 1d 07 a9 c7 50 93 6d 31 42 6f
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f
yescrypt("", "", 1, 4, 1, 1, 1, 0) = 4b aa 8c d8 60 8b a9 1f 3e 34 39 d9 ec 4f ae 8f 9f c0 92 d9 ca 22 b7 37 7e 31 ae 5b 9a d7 87 7c 11 68 69 11 62 dd 0e 5e f0 49 e5 70 65 0c be d4 38 4a d6 05 34 fb 0c be d1 9f f3 f0 33 c9 4b 0c
yescrypt("", "", 1, 4, 1, 1, 2, 0) = e6 e8 bb a0 9b 64 12 ff b0 b3 cc 35 e3 7d 0b 78 2a 47 fb aa dc 57 a0 76 d7 c6 cc 2e 70 91 9a 1b 8d 47 38 c4 f8 33 55 69 07 42 d9 be d7 1c 3b 8f b0 d7 eb 08 6a b1 34 c5 e5 57 07 c2 c1 3c 75 ef
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99 7d 70 ae 0a 64 bf 0a 4d 96 c1 73 ab f8 82 79 c1 a9 4a d9 bd f1 68 ed fb bd 90 f6 6e d5 c8 0d
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76 eb 24 1d f8 11 9a 9a 12 2a e3 69 20 bc c7 f4 14 b9 c0 d5 8f 45 00 80 60 da de 46 b0 c8 09 22 bd cc 16 a3 ab 5d 20 1d 4c 61 40 c6 71 be 1f 75 27 2c a9 04 73 9d 5a d1 ff 67 2b 0c 21
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68 ca 36 2c 55 57 cc 04 b6 81 1e 2e 73 08 41 f5 26 d8 f4 f7 ac fb fa 9e 06 fe 1f 38 3a 71 15 5e
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23
yescrypt("p", "s", 182, 16, 8, 1, 10, 0) = e1 f9 81 73 3a 94 05 2f cd 7a cb 14 05 df 0b bd e8 e4 99 b6 a1 33 1b 77 59 09 b4 8c 2f 51 6c 40 dc c8 30 16 35 b7 23 7b
yescrypt("p", "s", 1, 16, 8, 1, 10, 0) = 9e 7a 40 97 64 42 84 cf 3b 73 b6 04 50 ff 23 0c dc b6 b1 b1 9b 15 09 ee b4 82 f6 96 c4 f1 c7 05 c0 0f 74 02 16 18 3a 12
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99 c3 d6 69 30 fb 20 23 bb
yescrypt("p", "s", 1, 16, 8, 1, 0, 0) = 9d d6 36 c2 d0 bb 92 34 52 86 ef da f8 a6 8c fc 1b 4f fd c4 b1 ad ac cc 7d 86 4b 9a 67 87 b8 5d 6a e0 f5 28 0d a8 88 9f
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91
'$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.'
Plaintext: '(null)'

real    0m3.479s
user    0m2.950s
sys 0m0.763s
$ diff -U0 TESTS-OK TESTS-OUT && echo PASSED || echo FAILED
--- TESTS-OK    2022-02-19 14:24:46.084345030 -0800
+++ TESTS-OUT   2022-02-20 12:14:47.233044310 -0800
@@ -4 +4 @@
-scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = 21 01 cb 9b 6a 51 1a ae ad db be 09 cf 70 f8 81 ec 56 8d 57 4a 2f fd 4d ab e5 ee 98 20 ad aa 47 8e 56 fd 8f 4b a5 d0 9f fa 1c 6d 92 7c 40 f4 c3 37 30 40 49 e8 a9 52 fb cb f4 5c 6f a7 7a 41 a4
+scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
@@ -31,50 +31 @@
-Plaintext: '$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.$HboGM6qPrsK.StKYGt6KErmUYtioHreJd98oIugoNB6'
-Encrypted: '$y$jD5.7$BkbiDbyWZnzlLWOAcru671$zLAHafRUyp9n9XZWnltUbj3ULWUtMN4fteTltjWkARC'
-Plaintext: '$y$jC4$LdJMENpBABJJ3hIHjB1B$jVg4HoqqpbmQv/NCpin.QCMagJ8o4QX7lXdzvVV0xFC'
-Encrypted: '$y$jC4$qiyh2SQgE5vrF3ORvFho$HurI7MuukXHz..TpxrwKuakji/j9VKDh2WVUK4DIsq5'
-Plaintext: '$y$/B3.6$LdJMENpBABJJ3hIHjB1$h8sE4hJo.BsdlfJr0.d8bNJNPZymH7Y3kLj4aY1Rfc8'
-Encrypted: '$y$/B3.6$YiN5s/dKpjNrdKm9ND0$lBNMoUaAsw.JR0zMq9IBKIi/VPxj7lD7Sg64nB5LFw2'
-Plaintext: '$y$/A2$LdJMENpBABJJ3hIHj/$5IEld1eWdmh5lylrqHLF5dvA3ISpimEM9J1Dd05n/.3'
-Encrypted: '$y$/A2$TqjvmGXoqnsNU/8Y40$ZB43..7UnMt6ySU7XbpPyvyahahHSkJJgztkLWp6/IC'
-Plaintext: '$y$j91.5$LdJMENpBABJJ3hIH$ebKnn23URD5vyLgF9cP2EvVosrUXf7UErGRV0KmC6e6'
-Encrypted: '$y$j91.5$bICydADAaInC9UR/$rpsDDkULkr1caCVYV9PNAsiZxijoQ2/gDcbonlowRi7'
-Plaintext: '$y$j80$LdJMENpBABJJ3h2$ysXVVJwuaVlI1BWoEKt/Bz3WNDDmdOWz/8KTQaHL1cC'
-Encrypted: '$y$j80$fmxyeGYOC34lh19$mm9FRBs0iHLTHfSNznm1kJVchXN4PaS8hoNI6TTAlB7'
-Plaintext: '$y$/7/.4$LdJMENpBABJJ3/$lXHleh7bIZMGNtJVxGVrsIWkEIXfBedlfPui/PITflC'
-Encrypted: '$y$/7/.4$EuTbL3Wtm3khW0$1jvKQzIcAqYnYxfb4TEs.FeAZ7rLDl5vNQEkPPcj2KC'
-Plaintext: '$y$/6.$LdJMENpBABJJ$zQITmYSih5.CTY47x0IuE4wl.b3HzYGKKCSggakaQ22'
-Encrypted: '$y$/6.$x0G/jIr053ui$4O.aVGTPptkjx6eXrW8fdvqcPEV28w7a1PSos6CXV31'
-Plaintext: '$y$j5..3$LdJMENpBAB3$xi27PTUNd8NsChHeLOz85JFnUOyibRHkWzprowRlR5/'
-Encrypted: '$y$j5..3$/nwg3UXJWp/$5jcvDgeotKpaG9IeSJx0fJNSz33JjTYYD4Kwao3Eki5'
-Plaintext: '$y$j4/$LdJMENpBA/$tHlkpTQ8V/eEnTVau1uW36T97LIXlfPrEzdeV5SE5K7'
-Encrypted: '$y$j4/$yoneNBwae0$uPBnH0yXBCOM5v5BU9qlvUUtUr3QD5btS0upc6sdvf4'
-Plaintext: '$y$/3..2$LdJMENpB$tNczXFuNUd3HMqypStCRsEaL4e4KF7ZYLBe8Hbeg0B7'
-Encrypted: '$y$/3..2$VD20uHT3$AV5WWaN6bEKRvZlCuurj.mnHMmZmJ9ExQ9HjiReCDwC'
-Plaintext: '$y$/2/$LdJMEN3$RRorHhfsw1/P/WR6Aurg4U72e9Q7qt9vFPURdyfiqK8'
-Encrypted: '$y$/2/$BYujKJA$fsMwVvFm8r1caFQP.mem3OUuMYBCDGj9CEoDfSwFDLB'
-Plaintext: '$y$j2..1$LdJME/$iLEt6kuTwHch6XdCxtTHfsQzYwWFmpUwgl6Ax8RH4d1'
-Encrypted: '$y$j2..1$.mZga/$X6GFMkoYPxFapo.3H4LllEjltFapONQcKUOdEd9oPa/'
-Plaintext: '$y$j0/$LdJM$k7BXzSDuoGHW56SY3HxROCiA0gWRscZe2aA0q5oHPM0'
-Encrypted: '$y$j0/$SkNZ$DQ06H0br45bpE7lGgCD9gOxTMP9SsO6Mt1T9lo5PHz1'
-Plaintext: '$y$//..0$Ld3$6BJXezMFxaMiO5wsuoEmztvtCs/79085dZO56ADlV5B'
-Encrypted: '$y$//..0$lM1$60gjeUIW/3QidfN6zU9NqB09Ni1NBMfj2VaSZMjDd18'
-Plaintext: '$y$///$L/$Rrrkp6OVljrIk0kcwkCDhAiHJiSthh3cKeIGHUW7Z0C'
-Encrypted: '$y$///$q.$/.tR4GqigxciLYGoB8fmzudWQR7IzSu9s3dR8wp3VsD'
-Plaintext: '$y$j1../$LdJMENpBABJJ3hIHjB1Bi.$L8OQFc8mxJPd7CpUFgkS7KqJM2I9jGXu3BdqX2D.647'
-Encrypted: '$y$j1../$BkbiDbyWZnzlLWOAcru671$iicGI2gNZyhimPVgz2VoKrJAB9fWykBN.3Mh0AwEy29'
-Plaintext: '$y$j//$LdJMENpBABJJ3hIHjB1B$U8a2MaK.yesqWySK8Owk6PWeWmp/XuagMbpP45q1/q1'
-Encrypted: '$y$j//$qiyh2SQgE5vrF3ORvFho$5dD9ick8ugystfp8wa3xbV7ASDux0dpoOh0QJxFuXH5'
-'$7$C6..../....SodiumChloride$kBGj9fHznVYFQMEn/qDCfrDevf9YDtcDdKvEqHJLV8D'
-'$7$06..../....SodiumChloride$ENlyo6fGw4PCcDBOFepfSZjFUnVatHzCcW55.ZGz3B0'
-r=8 N=2^11 NROM=2^18
-Will use 262144.00 KiB ROM
-         2048.00 KiB RAM
-Initializing ROM ... DONE (696ebab2)
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.'
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
-Initializing ROM in preallocated memory ... DONE (696ebab2)
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
-'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$K3wFVK9/t3QsjCk/oK2s8dKzzZ4m7QTP8Ms5uywhWv8'
-'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$/OnmIkP0UK5OxyxD0Af/V1oL0zWvTLAUWg3Nr0bsFEB'
-'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$DskEGULspNduIZVFK5SOK8enlXnSs/vkuXFdi0wkQ1.'
-'$y$j/.5I$LdJMENpBABJJ3hIHjB1Bi.$NqCMKxN9Y9Uw821.72ScGDMpyk7U7V51qnHSRPapzW8'
+Plaintext: '(null)'
FAILED

Second machine, default config:

$ time ./tests | tee TESTS-OUT
scrypt("", "", 16, 1, 1) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
scrypt("password", "NaCl", 1024, 8, 16) = fd ba be 1c 9d 34 72 00 78 56 e7 19 0d 01 e9 fe 7c 6a d7 cb c8 23 78 30 e7 73 76 63 4b 37 31 62 2e af 30 d9 2e 22 a3 88 6f f1 09 27 9d 98 30 da c7 27 af b9 4a 83 ee 6d 83 60 cb df a2 cc 06 40
scrypt("pleaseletmein", "SodiumChloride", 16384, 8, 1) = 70 23 bd cb 3a fd 73 48 46 1c 06 cd 81 fd 38 eb fd a8 fb ba 90 4f 8e 3e a9 b5 43 f6 54 5d a1 f2 d5 43 29 55 61 3f 0f cf 62 d4 97 05 24 2a 9a f9 e6 1e 85 dc 0d 65 1e 40 df cf 01 7b 45 57 58 87
scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = 21 01 cb 9b 6a 51 1a ae ad db be 09 cf 70 f8 81 ec 56 8d 57 4a 2f fd 4d ab e5 ee 98 20 ad aa 47 8e 56 fd 8f 4b a5 d0 9f fa 1c 6d 92 7c 40 f4 c3 37 30 40 49 e8 a9 52 fb cb f4 5c 6f a7 7a 41 a4
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20 3b 19 ca 42 c1 8a 04 97 f1 6b 48 44 e3 07 4a e8 df df fa 3f ed e2 14 42 fc d0 06 9d ed 09 48 f8 32 6a 75 3a 0f c8 1f 17 e8 d3 e0 fb 2e 0d 36 28 cf 35 e2 0c 38 d1 89 06
yescrypt("", "", 0, 16, 1, 1, 0, 0) = 77 d6 57 62 38 65 7b 20
yescrypt("", "", 0, 4, 1, 1, 0, 0) = ef ad 0c 23 31 4c b5 72 bc 3c fb 15 43 da 42 f8 a8 b0 73 00 4c 86 6b 64 ab 50 55 a4 f0 9f a5 f5 71 14 2e bf e7 e0 5a 3b 92 c4 32 f3 1d ea 95 ad 5f 9c 85 4b 64 56 46 2f 4b d0 f7 32 b7 cd c5 49
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f 7f 1a e8 b4 df ed a5 1f 8b 6d 56 f3 08 1b e1 a7 c0 83 3b a2 71 9a 36 ab 02 88 5d ae 36 55 7d 34 26 86 b1 7b a7 5f 2c 21 77 92 de 09 70 ab 1d 07 a9 c7 50 93 6d 31 42 6f
yescrypt("", "", 1, 4, 1, 1, 0, 0) = 85 dd a4 8c 9e c9 de 2f
yescrypt("", "", 1, 4, 1, 1, 1, 0) = 4b aa 8c d8 60 8b a9 1f 3e 34 39 d9 ec 4f ae 8f 9f c0 92 d9 ca 22 b7 37 7e 31 ae 5b 9a d7 87 7c 11 68 69 11 62 dd 0e 5e f0 49 e5 70 65 0c be d4 38 4a d6 05 34 fb 0c be d1 9f f3 f0 33 c9 4b 0c
yescrypt("", "", 1, 4, 1, 1, 2, 0) = e6 e8 bb a0 9b 64 12 ff b0 b3 cc 35 e3 7d 0b 78 2a 47 fb aa dc 57 a0 76 d7 c6 cc 2e 70 91 9a 1b 8d 47 38 c4 f8 33 55 69 07 42 d9 be d7 1c 3b 8f b0 d7 eb 08 6a b1 34 c5 e5 57 07 c2 c1 3c 75 ef
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99 7d 70 ae 0a 64 bf 0a 4d 96 c1 73 ab f8 82 79 c1 a9 4a d9 bd f1 68 ed fb bd 90 f6 6e d5 c8 0d
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08 99
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1 08
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac d9 a4 20 1c f4 a4 76 ec f7 ba a6 11 3d 86 fb 65 cd 07 10 2b 40 04 e4 f9 d9 9c d3 42 55 a1
yescrypt("", "", 1, 4, 1, 1, 3, 0) = ac
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76 eb 24 1d f8 11 9a 9a 12 2a e3 69 20 bc c7 f4 14 b9 c0 d5 8f 45 00 80 60 da de 46 b0 c8 09 22 bd cc 16 a3 ab 5d 20 1d 4c 61 40 c6 71 be 1f 75 27 2c a9 04 73 9d 5a d1 ff 67 2b 0c 21
yescrypt("", "", 182, 4, 1, 1, 0, 0) = 0c d5 af 76
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68 ca 36 2c 55 57 cc 04 b6 81 1e 2e 73 08 41 f5 26 d8 f4 f7 ac fb fa 9e 06 fe 1f 38 3a 71 15 5e
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4 68
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35 c4
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23 b6 ad f0 b6 0c 9a 99 7f 58 58 3d 80 cd a4 8c 63 8c dc 2f 28 9e df 93 a7 08 07 72 5a 0d 35
yescrypt("", "", 182, 4, 1, 1, 1, 0) = 23
yescrypt("p", "s", 182, 16, 8, 1, 10, 0) = e1 f9 81 73 3a 94 05 2f cd 7a cb 14 05 df 0b bd e8 e4 99 b6 a1 33 1b 77 59 09 b4 8c 2f 51 6c 40 dc c8 30 16 35 b7 23 7b
yescrypt("p", "s", 1, 16, 8, 1, 10, 0) = 9e 7a 40 97 64 42 84 cf 3b 73 b6 04 50 ff 23 0c dc b6 b1 b1 9b 15 09 ee b4 82 f6 96 c4 f1 c7 05 c0 0f 74 02 16 18 3a 12
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99 c3 d6 69 30 fb 20 23 bb
yescrypt("p", "s", 1, 16, 8, 1, 0, 0) = 9d d6 36 c2 d0 bb 92 34 52 86 ef da f8 a6 8c fc 1b 4f fd c4 b1 ad ac cc 7d 86 4b 9a 67 87 b8 5d 6a e0 f5 28 0d a8 88 9f
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91 c3 f2 60 89 48 78 2c d6 89 cc 45 57 90 17 aa a5 ff 8b aa 74 a6 32 ec 99
yescrypt("p", "s", 182, 16, 8, 1, 0, 0) = c8 c7 ff 11 22 b0 b2 91
'$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.'
Plaintext: '$y$jD5.7$LdJMENpBABJJ3hIHjB1Bi.$HboGM6qPrsK.StKYGt6KErmUYtioHreJd98oIugoNB6'
Encrypted: '$y$jD5.7$BkbiDbyWZnzlLWOAcru671$zLAHafRUyp9n9XZWnltUbj3ULWUtMN4fteTltjWkARC'
Plaintext: '$y$jC4$LdJMENpBABJJ3hIHjB1B$jVg4HoqqpbmQv/NCpin.QCMagJ8o4QX7lXdzvVV0xFC'
Encrypted: '$y$jC4$qiyh2SQgE5vrF3ORvFho$HurI7MuukXHz..TpxrwKuakji/j9VKDh2WVUK4DIsq5'
Plaintext: '$y$/B3.6$LdJMENpBABJJ3hIHjB1$h8sE4hJo.BsdlfJr0.d8bNJNPZymH7Y3kLj4aY1Rfc8'
Encrypted: '$y$/B3.6$YiN5s/dKpjNrdKm9ND0$lBNMoUaAsw.JR0zMq9IBKIi/VPxj7lD7Sg64nB5LFw2'
Plaintext: '$y$/A2$LdJMENpBABJJ3hIHj/$5IEld1eWdmh5lylrqHLF5dvA3ISpimEM9J1Dd05n/.3'
Encrypted: '$y$/A2$TqjvmGXoqnsNU/8Y40$ZB43..7UnMt6ySU7XbpPyvyahahHSkJJgztkLWp6/IC'
Plaintext: '$y$j91.5$LdJMENpBABJJ3hIH$ebKnn23URD5vyLgF9cP2EvVosrUXf7UErGRV0KmC6e6'
Encrypted: '$y$j91.5$bICydADAaInC9UR/$rpsDDkULkr1caCVYV9PNAsiZxijoQ2/gDcbonlowRi7'
Plaintext: '$y$j80$LdJMENpBABJJ3h2$ysXVVJwuaVlI1BWoEKt/Bz3WNDDmdOWz/8KTQaHL1cC'
Encrypted: '$y$j80$fmxyeGYOC34lh19$mm9FRBs0iHLTHfSNznm1kJVchXN4PaS8hoNI6TTAlB7'
Plaintext: '$y$/7/.4$LdJMENpBABJJ3/$lXHleh7bIZMGNtJVxGVrsIWkEIXfBedlfPui/PITflC'
Encrypted: '$y$/7/.4$EuTbL3Wtm3khW0$1jvKQzIcAqYnYxfb4TEs.FeAZ7rLDl5vNQEkPPcj2KC'
Plaintext: '$y$/6.$LdJMENpBABJJ$zQITmYSih5.CTY47x0IuE4wl.b3HzYGKKCSggakaQ22'
Encrypted: '$y$/6.$x0G/jIr053ui$4O.aVGTPptkjx6eXrW8fdvqcPEV28w7a1PSos6CXV31'
Plaintext: '$y$j5..3$LdJMENpBAB3$xi27PTUNd8NsChHeLOz85JFnUOyibRHkWzprowRlR5/'
Encrypted: '$y$j5..3$/nwg3UXJWp/$5jcvDgeotKpaG9IeSJx0fJNSz33JjTYYD4Kwao3Eki5'
Plaintext: '$y$j4/$LdJMENpBA/$tHlkpTQ8V/eEnTVau1uW36T97LIXlfPrEzdeV5SE5K7'
Encrypted: '$y$j4/$yoneNBwae0$uPBnH0yXBCOM5v5BU9qlvUUtUr3QD5btS0upc6sdvf4'
Plaintext: '$y$/3..2$LdJMENpB$tNczXFuNUd3HMqypStCRsEaL4e4KF7ZYLBe8Hbeg0B7'
Encrypted: '$y$/3..2$VD20uHT3$AV5WWaN6bEKRvZlCuurj.mnHMmZmJ9ExQ9HjiReCDwC'
Plaintext: '$y$/2/$LdJMEN3$RRorHhfsw1/P/WR6Aurg4U72e9Q7qt9vFPURdyfiqK8'
Encrypted: '$y$/2/$BYujKJA$fsMwVvFm8r1caFQP.mem3OUuMYBCDGj9CEoDfSwFDLB'
Plaintext: '$y$j2..1$LdJME/$iLEt6kuTwHch6XdCxtTHfsQzYwWFmpUwgl6Ax8RH4d1'
Encrypted: '$y$j2..1$.mZga/$X6GFMkoYPxFapo.3H4LllEjltFapONQcKUOdEd9oPa/'
Plaintext: '$y$j0/$LdJM$k7BXzSDuoGHW56SY3HxROCiA0gWRscZe2aA0q5oHPM0'
Encrypted: '$y$j0/$SkNZ$DQ06H0br45bpE7lGgCD9gOxTMP9SsO6Mt1T9lo5PHz1'
Plaintext: '$y$//..0$Ld3$6BJXezMFxaMiO5wsuoEmztvtCs/79085dZO56ADlV5B'
Encrypted: '$y$//..0$lM1$60gjeUIW/3QidfN6zU9NqB09Ni1NBMfj2VaSZMjDd18'
Plaintext: '$y$///$L/$Rrrkp6OVljrIk0kcwkCDhAiHJiSthh3cKeIGHUW7Z0C'
Encrypted: '$y$///$q.$/.tR4GqigxciLYGoB8fmzudWQR7IzSu9s3dR8wp3VsD'
Plaintext: '$y$j1../$LdJMENpBABJJ3hIHjB1Bi.$L8OQFc8mxJPd7CpUFgkS7KqJM2I9jGXu3BdqX2D.647'
Encrypted: '$y$j1../$BkbiDbyWZnzlLWOAcru671$iicGI2gNZyhimPVgz2VoKrJAB9fWykBN.3Mh0AwEy29'
Plaintext: '$y$j//$LdJMENpBABJJ3hIHjB1B$U8a2MaK.yesqWySK8Owk6PWeWmp/XuagMbpP45q1/q1'
Encrypted: '$y$j//$qiyh2SQgE5vrF3ORvFho$5dD9ick8ugystfp8wa3xbV7ASDux0dpoOh0QJxFuXH5'
'$7$C6..../....SodiumChloride$kBGj9fHznVYFQMEn/qDCfrDevf9YDtcDdKvEqHJLV8D'
'$7$06..../....SodiumChloride$ENlyo6fGw4PCcDBOFepfSZjFUnVatHzCcW55.ZGz3B0'
r=8 N=2^11 NROM=2^18
Will use 262144.00 KiB ROM
         2048.00 KiB RAM
Initializing ROM ... DONE (696ebab2)
'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.'
'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
Initializing ROM in preallocated memory ... DONE (696ebab2)
'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$4XJGTsv75AjIN60Z31kPN3.86vkCYzIq7LMz2Pb2lC.'
'$y$j8567F$LdJMENpBABJJ3hIHjB1Bi.$K3wFVK9/t3QsjCk/oK2s8dKzzZ4m7QTP8Ms5uywhWv8'
'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$/OnmIkP0UK5OxyxD0Af/V1oL0zWvTLAUWg3Nr0bsFEB'
'$y$j8567F$LdJMENpBABJJ3ZIHjB1Bi.$DskEGULspNduIZVFK5SOK8enlXnSs/vkuXFdi0wkQ1.'
'$y$j/.5I$LdJMENpBABJJ3hIHjB1Bi.$NqCMKxN9Y9Uw821.72ScGDMpyk7U7V51qnHSRPapzW8'

real    0m4.455s
user    0m9.636s
sys 0m1.466s
$ diff -U0 TESTS-OK TESTS-OUT && echo PASSED || echo FAILED
PASSED

Looks like I misread the diff output as the test output, and the test was simply not run as time was not found, for some reason. So, whatever is going on, it's still a problem under kernel version 4.15.0.

alexforencich commented 2 years ago

And under strace, we can compare what happens.

Defaults:

mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 1073744896, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc90ffff000
futex(0x5619d7970d94, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x5619d7970d94, FUTEX_WAKE_PRIVATE, 2147483647) = 0
munmap(0x7fc90ffff000, 1073744896)      = 0
write(1, "scrypt(\"pleaseletmein\", \"SodiumC"..., 251scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = 21 01 cb 9b 6a 51 1a ae ad db be 09 cf 70 f8 81 ec 56 8d 57 4a 2f fd 4d ab e5 ee 98 20 ad aa 47 8e 56 fd 8f 4b a5 d0 9f fa 1c 6d 92 7c 40 f4 c3 37 30 40 49 e8 a9 52 fb cb f4 5c 6f a7 7a 41 a4
) = 251

1G huge pages:

mmap(NULL, 1075838976, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0) = 0x7fa880000000
futex(0x558755ca3d94, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x558755ca3d94, FUTEX_WAKE_PRIVATE, 2147483647) = 0
munmap(0x7fa880000000, 1075838976)      = -1 EINVAL (Invalid argument)
write(1, "scrypt(\"pleaseletmein\", \"SodiumC"..., 66scrypt("pleaseletmein", "SodiumChloride", 1048576, 8, 1) = FAILED
) = 66
alexforencich commented 2 years ago

Hmm, seems like this is a documented "feature" https://man7.org/linux/man-pages/man2/mmap.2.html

For mmap, length is internally rounded up to the huge page size, but for munmap, this does not happen and the length needs to be correct. So, what's the proper solution? Seems like mmap supports MAP_HUGE_2MB and MAP_HUGE_1GB, so I think using this may be the "best" solution here...if you're going to assume a 2 MB huge page size, just force mmap to use only 2 MB huge pages, otherwise fail.

I did a quick test adding MAP_HUGE_2MB alongside MAP_HUGETLB, and the tests passed as the first call to mmap failed due to no 2 MB huge pages being available (FYI, I also needed to include linux/mman.h for MAP_HUGE_2MB).

But, another consideration may be to drop explicit support for huge pages altogether and instead use transparent hugepages by using madvise with MADV_HUGEPAGE. See https://www.kernel.org/doc/Documentation/vm/transhuge.txt .

solardiz commented 2 years ago

Regarding time, the one you tested manually is bash built-in. Maybe your /bin/sh (which is what make runs commands under) isn't bash and you do not have the separate time package/program installed. Anyway, maybe we should drop usage of time from those tests.

Thank you for figuring out that the mmap vs. munmap discrepancy in required rounding of size is actually documented, and for suggesting these alternatives. I think I'll try MAP_HUGE_2MB and possibly also MADV_HUGEPAGE in the fallback, although since the allocation is typically rather short-lived I'm concerned that on-the-fly conversion to huge pages (perhaps involving data copying) might cost more performance than subsequent usage of those pages would save. Also, when the allocation address isn't huge page aligned, I suppose the conversion would only be possible for pages in the middle of the allocation. Ideally, the kernel would be allocating huge pages right away (at least for those middle pages), but I'm not sure it will even with the advise (but maybe!)

solardiz commented 2 years ago

Trying MAP_HUGETLB on an outdated RHEL7'ish kernel now, with some huge pages pre-allocated with sysctl vm.nr_hugepages=1000, I got a kernel crash (in region_chg called from vma_needs_reservation called from alloc_huge_page). So this stuff is fragile. Luckily, this only affects configurations that explicitly pre-allocated huge pages (otherwise the first mmap fails and we get the fallback).

On the same old system, MADV_HUGEPAGE returns success, with no apparent effect on performance (and no crash either).

alexforencich commented 2 years ago

Oh, that would make sense, I think ubuntu uses dash for sh instead of bash. Didn't realize time was a bash built-in.

Possibly hugepages vs transparent huge pages vs normal pages is worth some benchmarks. Seems like transparent huge pages should be available on most systems since the default setting seems to be madvise. I'm not sure how to check what's taking place in kernel in terms of actually allocating huge pages after you've used madvise(..., MADV_HUGEPAGE). Possibly this can be checked in /proc/self/pagemap. Unfortunately, I think this is managed separately by khugepaged, so possibly the mappings could change independently of what the program is doing, although I would hope that it does something sensible for a fresh allocation that has yet to be accessed. Hugepages can only be used when they are explicitly configured, and I'm not sure how common that is because they're not swappable and the space reserved for hugepages cannot be used for anything else. The other thing that I noticed is that it seems like you can only configure one size of huge pages at a time (either 2 MB or 1 GB). But if you configure 1 GB huge pages, you can still use 2 MB transparent huge pages. Although, on systems that have huge pages configured, it seems like there might be a good chance that transparent huge pages are disabled, so it probably makes sense to support both.

Mic92 commented 1 year ago

Would it make sense to disable hugetable support until these issues are sorted out?

solardiz commented 1 year ago

@Mic92 Yes, I'm considering either dropping explicit huge pages support or using MAP_HUGE_2MB (where available, and excluding explicit usage of huge pages otherwise).

solardiz commented 1 year ago

So I was thinking that maybe on recent kernels explicit huge page support was no longer helpful - but no, on Fedora 36 with a 5.18'ish kernel I am getting +4% speed at one hash computation, +3% at many concurrent computations for 1 GiB per hash allocation (useful for the KDF use case of upstream yescrypt, and not so much for libxcrypt's password hashing use case). To achieve these speedups, I have to pre-allocate huge pages with the vm.nr_hugepages sysctl setting... which I think most systems won't, though.

Anyway, the below patch retains those speedups in that special case while hopefully fixing the issue reported here:

+++ b/yescrypt-platform.c
@@ -21,6 +21,9 @@
 #ifdef __unix__
 #include <sys/mman.h>
 #endif
+#ifdef __linux__
+#include <linux/mman.h> /* for MAP_HUGE_2MB */
+#endif

 #define HUGEPAGE_THRESHOLD             (32 * 1024 * 1024)

@@ -40,11 +43,11 @@ static void *alloc_region(yescrypt_region_t *region, size_t size)
            MAP_NOCORE |
 #endif
            MAP_ANON | MAP_PRIVATE;
-#if defined(MAP_HUGETLB) && defined(HUGEPAGE_SIZE)
+#if defined(MAP_HUGETLB) && defined(MAP_HUGE_2MB) && defined(HUGEPAGE_SIZE)
        size_t new_size = size;
        const size_t hugepage_mask = (size_t)HUGEPAGE_SIZE - 1;
        if (size >= HUGEPAGE_THRESHOLD && size + hugepage_mask >= size) {
-               flags |= MAP_HUGETLB;
+               flags |= MAP_HUGETLB | MAP_HUGE_2MB;
 /*
  * Linux's munmap() fails on MAP_HUGETLB mappings if size is not a multiple of
  * huge page size, so let's round up to huge page size here.
@@ -56,7 +59,7 @@ static void *alloc_region(yescrypt_region_t *region, size_t size)
        if (base != MAP_FAILED) {
                base_size = new_size;
        } else if (flags & MAP_HUGETLB) {
-               flags &= ~MAP_HUGETLB;
+               flags &= ~(MAP_HUGETLB | MAP_HUGE_2MB);
                base = mmap(NULL, size, PROT_READ | PROT_WRITE, flags, -1, 0);
        }

Its downside is the minimum kernel version for explicit usage of huge pages is increased from 2.6.32 (RHEL6) to 3.8 (RHEL7). IIRC, I had previously found them making most performance difference on RHEL6 (or its rebuild distros), so that's kind of a pity, but then that distro is EOL'ed.

I think I'll get the above patch into upstream yescrypt, and libxcrypt can do the same to minimize the differences.

solardiz commented 1 year ago

on Fedora 36 with a 5.18'ish kernel I am getting +4% speed at one hash computation, +3% at many concurrent computations for 1 GiB per hash allocation

There's also a 1% speedup at 16 MiB (which we currently typically use for password hashing with libxcrypt), but the threshold is set at 32 MiB. Initially (prior to yescrypt 1.0 release), I had the threshold at 12 MiB (based on testing back then). The increase to 32 MiB was to better accommodate concurrent RAM+ROM usage, where having only the ROM on huge pages (typically SysV shm) resulted in slight speedup on some systems compared to keeping both RAM and ROM on huge pages - I guess due to use of different TLBs vs. competition for the one same huge page TLB. yespower - a related PoW scheme, which doesn't have the ROM - has the threshold at 12 MiB.

Since libxcrypt does not expose yescrypt's ROM support, it could also lower the threshold to 12 MiB once we fix the issue here. A drawback is that it'd actually consume almost 2 MiB of extra RAM (but only out of pre-allocated huge pages, so OK?) since the typical allocation is just slightly higher than 16 MiB (so it'd end up with 18 MiB).

Mic92 commented 1 year ago

Thanks. I just applied the patch and it works for both 1GiB and 2MiB pages on a system that used to fail before.

solardiz commented 1 year ago

Thanks for testing. I've just pushed this fix to upstream yescrypt: https://github.com/openwall/yescrypt/commit/1554b087ffe2d8c5c07c2795fe5cc21558b8a036

Mic92 commented 1 year ago

Could we get a patch release for nixpkgs? https://github.com/NixOS/nixpkgs/pull/201653

mweinelt commented 1 year ago

cc @besser82 ^

besser82 commented 1 year ago

@Mic92, @mweinelt: Fixed in release v4.4.32.