search

gost-engine / engine

A reference implementation of the Russian GOST crypto algorithms for OpenSSL
Apache License 2.0
376 stars 170 forks source link

current master fails to configure - libprov dir is empty #359

Open mouse07410 opened 3 years ago

mouse07410 commented 3 years ago

MacOS Big Sur 11.6, Xcode-13.0 (both factors irrelevant, IMHO).

+ LDFLAGS=
+ OPENSSL_DIR=/Users/uri/openssl-3
+ OPENSSL_ENGINES_DIR=/Users/uri/openssl-3/lib/engines-3
+ THREE=-3-
+ '[' -z clang ']'
+ '[' -z '' ']'
+ export OPENSSL_ROOT_DIR=/Users/uri/openssl-3
+ OPENSSL_ROOT_DIR=/Users/uri/openssl-3
+ '[' -z ']'
+ CMAKE_BUILD_TYPE=Release
+ OPENSSL_INCLUDE_DIR=/Users/uri/openssl-3/include
+ OPENSSL_CRYPTO_LIBRARY=/Users/uri/openssl-3/lib/libcrypto.dylib
+ OPENSSL_SSL_LIBRARY=/Users/uri/openssl-3/lib/libssl.dylib
+ PKG_CONFIG_PATH=/Users/uri/openssl-3/lib/pkgconfig:/opt/local/lib/pkgconfig:/opt/local/sha
re/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib/pkgconfig
+ OPENSSL_CFLAGS='-march=native -std=gnu17'
+ OPENSSL_LIB_DIR=/Users/uri/openssl-3/lib
+ OPENSSL_CONF=/Users/uri/openssl-3/etc/openssl.cnf
+ rm -rf build
+ mkdir -p build
+ cd build
+ cmake .. -DCMAKE_C_COMPILER=clang -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE:BOOL
=ON -DOPENSSL_ROOT_DIR=/Users/uri/openssl-3 -DOPENSSL_ENGINES_DIR=/Users/uri/openssl-3/lib/e
ngines-3
-- The C compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/local/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found OpenSSL: /Users/uri/openssl-3/lib/libcrypto.dylib (found suitable version "3.1.0", minimum required is "3.0")  
-- Found OpenSSL ...
CMAKE_BUILD_TYPE = Release
-- OpenSSL version values were not properly defined (OpenSSL-3 issue), setting to 3.0
-- Found OpenSSL includes:   /Users/uri/openssl-3/include
-- Found OpenSSL crypto lib: /Users/uri/openssl-3/lib/libcrypto.dylib
-- Found OpenSSL application: 
-- Found OpenSSL application: /Users/uri/openssl-3/bin/openssl
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - not found
-- LITTLE_ENDIAN
-- Performing Test ADDCARRY_U64
-- Performing Test ADDCARRY_U64 - Success
-- Performing Test RELAXED_ALIGNMENT
-- Performing Test RELAXED_ALIGNMENT - Success
-- No Test2::V0 perl module (engine and provider tests skipped)
CMake Error at CMakeLists.txt:376 (add_subdirectory):
  The source directory

    /Users/uri/src/grasshopper-engine/libprov

  does not contain a CMakeLists.txt file.

INSTALL TARGETS - target lib_gost_engine has PUBLIC_HEADER files but no PUBLIC_HEADER DESTINATION.
-- Configuring incomplete, errors occurred!

See also "/Users/uri/src/grasshopper-engine/build/CMakeFiles/CMakeOutput.log".
See also "/Users/uri/src/grasshopper-engine/build/CMakeFiles/CMakeError.log".
beldmit commented 3 years ago

You should perform git submodule update --init to fetch the libprov submodule

mouse07410 commented 3 years ago

Update

After pulling libprov submodule, master configures and builds - but fails keyexpimp test:

.  .  .  .  .
test 12
      Start 12: keyexpimp

12: Test command: /Users/uri/src/grasshopper-engine/build/bin/test_keyexpimp
12: Test timeout computed to be: 10000000
12/13 Test #12: keyexpimp ........................***Exception: SegFault  0.01 sec
.  .  .  .  .

Crash report:

Process:               test_keyexpimp [54239]
Path:                  /Users/USER/*/test_keyexpimp
Identifier:            test_keyexpimp
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        ctest [54209]
Responsible:           Terminal [583]
User ID:               501

Date/Time:             2021-10-14 01:11:28.698 -0400
OS Version:            macOS 11.6 (20G165)
Report Version:        12
Anonymous UUID:        B3544DF6-AE65-3AD0-0E52-DD6C16797F5A

Sleep/Wake UUID:       73CBEBA0-C23E-4C74-B7C6-9C1A282421FA

Time Awake Since Boot: 1000000 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x00000000000004a6
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [54239]

VM Regions Near 0x4a6:
--> 
    __TEXT                      1004a6000-1004aa000    [   16K] r-x/r-x SM=COW  /Users/*

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libcrypto.3.dylib               0x0000000100640b72 ossl_lh_strcasehash + 18 (lhash.c:370)
1   libcrypto.3.dylib               0x000000010066507d obj_name_hash + 61 (o_names.c:157)
2   libcrypto.3.dylib               0x0000000100640987 getrn + 8 (lhash.c:311) [inlined]
3   libcrypto.3.dylib               0x0000000100640987 OPENSSL_LH_retrieve + 39 (lhash.c:173)
4   libcrypto.3.dylib               0x0000000100664b16 lh_OBJ_NAME_retrieve + 5 (obj_local.h:12) [inlined]
5   libcrypto.3.dylib               0x0000000100664b16 OBJ_NAME_get + 118 (o_names.c:183)
6   libcrypto.3.dylib               0x000000010062a60c evp_get_cipherbyname_ex + 29 (names.c:85) [inlined]
7   libcrypto.3.dylib               0x000000010062a60c EVP_get_cipherbyname + 44 (names.c:72)
8   test_keyexpimp                  0x00000001004a82c8 omac_imit_ctrl + 616
9   test_keyexpimp                  0x00000001004a8865 gost_kexp15 + 261
10  test_keyexpimp                  0x00000001004a7b57 main + 423
11  libdyld.dylib                   0x00007fff20350f3d start + 1

When compiling engine with "-g", crash does not occur. It seems a compiler (Clang) bug, but a workaround would be nice, at least until Apple fixes it. I suspect it's related to -msse4.1, in case it matters.

beldmit commented 3 years ago

Don't know how to deal with it... Do you have something like valgrind on mac?

mouse07410 commented 3 years ago

Do you have something like valgrind on mac?

Alas, no. Might be able to use sanitizers, but...

On a different MacOS machine (the most powerful of my Mac herd ;), I'm getting Provider test failing on the master (and keyexpimp working). Current (as of today) OpenSSL 3.0 dev (aka, master).

15/15 Testing: provider
15/15 Test: provider
Command: "/opt/local/bin/prove" "--merge" "-PWrapOpenSSL" "/Users/ur20980/src/grasshopper-engine/test" "::" "provider"
Directory: /Users/ur20980/src/grasshopper-engine/build
"provider" start time: Oct 14 12:16 EDT
Output:
----------------------------------------------------------
Added /Users/ur20980/openssl-3/lib to:
  LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, LIBPATH
Added /Users/ur20980/openssl-3/bin to:
  PATH
LD_LIBRARY_PATH=/Users/ur20980/openssl-3/lib
DYLD_LIBRARY_PATH=/Users/ur20980/openssl-3/lib
LIBPATH=/Users/ur20980/openssl-3/lib
PATH=/Users/ur20980/openssl-3/bin:/opt/local/bin:/opt/local/sbin:/opt/local/libexec/gnubin:/opt/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin:/usr/local/go/bin:/usr/local/MacGPG2/bin:/Library/OpenSC/bin:/usr/local/share/dotnet:/opt/X11/bin:~/.dotnet/tools:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS:/Users/ur20980/.cargo/bin:/Users/ur20980/.cabal/bin:/Users/ur20980/.ghcup/bin:/Users/ur20980/.local/bin:/Users/ur20980/bin:/opt/local/libexec/qt5/bin:/Library/Java/NSS/bin:/Users/ur20980/go/bin:/opt/jruby/bin:/opt/local/Library/Frameworks/Python.framework/Versions/3.8/bin:/Users/ur20980/Library/Python/3.8/bin:/opt/GNAT/2019/bin:/Users/ur20980/src/OpenDDS/ACE_TAO/ACE/bin:/Users/ur20980/src/OpenDDS/bin
/Users/ur20980/src/grasshopper-engine/test/00-engine.t .... skipped: This test is only suitable for the engine
/Users/ur20980/src/grasshopper-engine/test/00-provider.t .. 
# Seeded srand with seed '20211014' from local date.
1..1
ok 1 - load provider without any config
ok
/Users/ur20980/src/grasshopper-engine/test/01-digest.t .... 
# Seeded srand with seed '20211014' from local date.
1..16
# Running tests for test type conf
ok 1 - GOST R 34.11-2012 256bit example 1 from standard
ok 2 - GOST R 34.11-2012 512bit example 1 from standard
ok 3 - GOST R 34.11-2012 256bit example 2 from standard
ok 4 - GOST R 34.11-2012 512bit example 2 from standard
ok 5 - GOST R 34.11-94 1K ascii
ok 6 - GOST R 34.11-2012 256bit 1K ascii
ok 7 - GOST R 34.11-2012 512bit 1K ascii
ok 8 - GOST R 34.11-94 1K binary
ok 9 - GOST R 34.11-2012 256bit 1K binary
ok 10 - GOST R 34.11-2012 512bit 1K binary
ok 11 - GOST R 34.11-94 539 bytes
ok 12 - GOST R 34.11-2012 256bit 539 bytes
ok 13 - GOST R 34.11-2012 512bit 539 bytes
ok 14 - GOST R 34.11-94 128K
ok 15 - GOST R 34.11-2012 256bit 128K
ok 16 - GOST R 34.11-2012 512bit 128K
ok
/Users/ur20980/src/grasshopper-engine/test/02-mac.t ....... 
# Seeded srand with seed '20211014' from local date.
1..19
# @ARGV = (provider)
# Running tests for test type conf
ok 1 - GOST MAC - default size
ok 2 - GOST MAC - size 1 bytes
ok 3 - GOST MAC - size 2 bytes
ok 4 - GOST MAC - size 3 bytes
ok 5 - GOST MAC - size 4 bytes
not ok 6 - GOST MAC - size 5 bytes

# Failed test 'GOST MAC - size 5 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 174.
# +--------------+----+--------------+
# | GOT          | OP | CHECK        |
# +--------------+----+--------------+
# | 2EE8D13D35\n | eq | 2EE8D13DFF\n |
# +--------------+----+--------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:5 -in testdata.dat gost-mac
not ok 7 - GOST MAC - size 6 bytes

# Failed test 'GOST MAC - size 6 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 174.
# +----------------+----+----------------+
# | GOT            | OP | CHECK          |
# +----------------+----+----------------+
# | 2EE8D13D3536\n | eq | 2EE8D13DFF7F\n |
# +----------------+----+----------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:6 -in testdata.dat gost-mac
not ok 8 - GOST MAC - size 7 bytes

# Failed test 'GOST MAC - size 7 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 174.
# +------------------+----+------------------+
# | GOT              | OP | CHECK            |
# +------------------+----+------------------+
# | 2EE8D13D353637\n | eq | 2EE8D13DFF7F03\n |
# +------------------+----+------------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:7 -in testdata.dat gost-mac
not ok 9 - GOST MAC - size 8 bytes

# Failed test 'GOST MAC - size 8 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 174.
# +--------------------+----+--------------------+
# | GOT                | OP | CHECK              |
# +--------------------+----+--------------------+
# | 2EE8D13D35363730\n | eq | 2EE8D13DFF7F037D\n |
# +--------------------+----+--------------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:8 -in testdata.dat gost-mac
ok 10 - GOST MAC - big data
ok 11 - GOST MAC parameters 2012 - default size
ok 12 - GOST MAC parameters 2012 - size 1 bytes
ok 13 - GOST MAC parameters 2012 - size 2 bytes
ok 14 - GOST MAC parameters 2012 - size 3 bytes
ok 15 - GOST MAC parameters 2012 - size 4 bytes
not ok 16 - GOST MAC parameters 2012 - size 5 bytes

# Failed test 'GOST MAC parameters 2012 - size 5 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 204.
# +--------------+----+--------------+
# | GOT          | OP | CHECK        |
# +--------------+----+--------------+
# | BE4453EC35\n | eq | BE4453EC1E\n |
# +--------------+----+--------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:5 -in testdata.dat gost-mac-12
not ok 17 - GOST MAC parameters 2012 - size 6 bytes

# Failed test 'GOST MAC parameters 2012 - size 6 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 204.
# +----------------+----+----------------+
# | GOT            | OP | CHECK          |
# +----------------+----+----------------+
# | BE4453EC3536\n | eq | BE4453EC1EC3\n |
# +----------------+----+----------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:6 -in testdata.dat gost-mac-12
not ok 18 - GOST MAC parameters 2012 - size 7 bytes

# Failed test 'GOST MAC parameters 2012 - size 7 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 204.
# +------------------+----+------------------+
# | GOT              | OP | CHECK            |
# +------------------+----+------------------+
# | BE4453EC353637\n | eq | BE4453EC1EC327\n |
# +------------------+----+------------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:7 -in testdata.dat gost-mac-12
not ok 19 - GOST MAC parameters 2012 - size 8 bytes

# Failed test 'GOST MAC parameters 2012 - size 8 bytes'
# at /Users/ur20980/src/grasshopper-engine/test/02-mac.t line 204.
# +--------------------+----+--------------------+
# | GOT                | OP | CHECK              |
# +--------------------+----+--------------------+
# | BE4453EC35363730\n | eq | BE4453EC1EC327BE\n |
# +--------------------+----+--------------------+
# Command was: openssl mac  -macopt key:0123456789abcdef0123456789abcdef -macopt size:8 -in testdata.dat gost-mac-12
Dubious, test returned 8 (wstat 2048, 0x800)
Failed 8/19 subtests 
/Users/ur20980/src/grasshopper-engine/test/03-encrypt.t ... 
# Seeded srand with seed '20211014' from local date.
1..48
# Running tests for test type conf
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 1 - CFB short text, paramset A - Trying to encrypt
ok 2 - CFB short text, paramset A - Checking that it encrypted correctly
ok 3 - CFB short text, paramset A - Trying to decrypt
ok 4 - CFB short text, paramset A - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 5 - CFB short text, paramset B - Trying to encrypt
ok 6 - CFB short text, paramset B - Checking that it encrypted correctly
ok 7 - CFB short text, paramset B - Trying to decrypt
ok 8 - CFB short text, paramset B - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 9 - CFB short text, paramset C - Trying to encrypt
ok 10 - CFB short text, paramset C - Checking that it encrypted correctly
ok 11 - CFB short text, paramset C - Trying to decrypt
ok 12 - CFB short text, paramset C - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 13 - CFB short text, paramset D - Trying to encrypt
ok 14 - CFB short text, paramset D - Checking that it encrypted correctly
ok 15 - CFB short text, paramset D - Trying to decrypt
ok 16 - CFB short text, paramset D - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 17 - CNT short text - Trying to encrypt
ok 18 - CNT short text - Checking that it encrypted correctly
ok 19 - CNT short text - Trying to decrypt
ok 20 - CNT short text - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 21 - CNT short text, paramset param doesnt affect cnt - Trying to encrypt
ok 22 - CNT short text, paramset param doesnt affect cnt - Checking that it encrypted correctly
ok 23 - CNT short text, paramset param doesnt affect cnt - Trying to decrypt
ok 24 - CNT short text, paramset param doesnt affect cnt - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 25 - CNT-12 short text - Trying to encrypt
ok 26 - CNT-12 short text - Checking that it encrypted correctly
ok 27 - CNT-12 short text - Trying to decrypt
ok 28 - CNT-12 short text - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 29 - CNT-12 short text, paramset param doesnt affect cnt - Trying to encrypt
ok 30 - CNT-12 short text, paramset param doesnt affect cnt - Checking that it encrypted correctly
ok 31 - CNT-12 short text, paramset param doesnt affect cnt - Trying to decrypt
ok 32 - CNT-12 short text, paramset param doesnt affect cnt - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 33 - CBC short text, paramset A - Trying to encrypt
ok 34 - CBC short text, paramset A - Checking that it encrypted correctly
ok 35 - CBC short text, paramset A - Trying to decrypt
ok 36 - CBC short text, paramset A - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 37 - CBC short text, paramset B - Trying to encrypt
ok 38 - CBC short text, paramset B - Checking that it encrypted correctly
ok 39 - CBC short text, paramset B - Trying to decrypt
ok 40 - CBC short text, paramset B - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 41 - CBC short text, paramset C - Trying to encrypt
ok 42 - CBC short text, paramset C - Checking that it encrypted correctly
ok 43 - CBC short text, paramset C - Trying to decrypt
ok 44 - CBC short text, paramset C - Checking that it decrypted correctly
Use of uninitialized value $test_type in concatenation (.) or string at /Users/ur20980/src/grasshopper-engine/test/03-encrypt.t line 81.
ok 45 - CBC short text, paramset D - Trying to encrypt
ok 46 - CBC short text, paramset D - Checking that it encrypted correctly
ok 47 - CBC short text, paramset D - Trying to decrypt
ok 48 - CBC short text, paramset D - Checking that it decrypted correctly
ok
/Users/ur20980/src/grasshopper-engine/test/04-pkey.t ...... skipped: TODO: add pkey support in provider

Test Summary Report
-------------------
/Users/ur20980/src/grasshopper-engine/test/02-mac.t     (Wstat: 2048 Tests: 19 Failed: 8)
  Failed tests:  6-9, 16-19
  Non-zero exit status: 8
Files=6, Tests=84,  1 wallclock secs ( 0.03 usr  0.02 sys +  0.68 cusr  0.36 csys =  1.09 CPU)
Result: FAIL
<end of output>
Test time =   1.19 sec
----------------------------------------------------------
Test Failed.
"provider" end time: Oct 14 12:16 EDT
"provider" time elapsed: 00:00:01
beldmit commented 3 years ago

@levitte, looks like a Mac-specific(?) problem with XOF...

levitte commented 3 years ago

Noted. I've seen something similar (not XOF related, though) in another provider I wrote. Maybe the reasons are similar... I'll have a look in a couple of days.

mouse07410 commented 3 years ago

I wonder if it's in any way related to the cause of this: https://github.com/randombit/botan/issues/2802 (compiler bug)?

levitte commented 3 years ago

It's a bit interesting, though, that the Github Actions don't fail. I wonder why that is...

Why I see on the provider test failure above is that the size macopt doesn't seem to go through all the way. The implementation itself defaults at 4:

https://github.com/gost-engine/engine/blob/df3ead272bd2019f98d16e6787f5df51556c0603/gost_crypt.c#L1213

... and my current working hypothesis is that somehow, the attempt to change that doesn't quite work:

https://github.com/gost-engine/engine/blob/df3ead272bd2019f98d16e6787f5df51556c0603/gost_prov_mac.c#L147

I wonder if this might be a type coersion error in the compiler. gctx->mac_size is a size_t, while EVP_MD_CTX_ctrl() takes an int in that position... so, guessing wildly here, could it be that arg inside EVP_MD_CTX_ctrl() has become zero? That would leave the XOF size unchanged (i.e. still 4), and get the exact (wrong) result seen above...

Does the result change with this simple patch?

diff --git a/gost_prov_mac.c b/gost_prov_mac.c
index ab04a9c..0948d37 100644
--- a/gost_prov_mac.c
+++ b/gost_prov_mac.c
@@ -144,7 +144,7 @@ static int mac_final(void *mctx, unsigned char *out, size_t *outl,
     if (out != NULL) {
         /* We ignore the error for GOST MDs that don't support setting
            the size */
-        EVP_MD_CTX_ctrl(gctx->dctx, EVP_MD_CTRL_XOF_LEN, gctx->mac_size, NULL);
+        EVP_MD_CTX_ctrl(gctx->dctx, EVP_MD_CTRL_XOF_LEN, (int)gctx->mac_size, NULL);
         ret = EVP_DigestFinal_ex(gctx->dctx, out, &tmpoutl);
     }
     if (outl != NULL)
beldmit commented 3 years ago

I wonder if it's in any way related to the cause of this: randombit/botan#2802 (compiler bug)?

I believe the test_keyexpimp failure may be related, but I thinkr @levitte's hypothesis is more relevant

mouse07410 commented 3 years ago

It's a bit interesting, though, that the Github Actions don't fail. I wonder why that is...

Because they don't use Clang with a CPU that has SSE4.1 enabled or detected? Or because their Clang is not v12?

mouse07410 commented 3 years ago

Does the result change with this simple patch?

I'm afraid not:

Running tests...
/opt/local/bin/ctest --force-new-ctest-process 
Test project /Users/ur20980/src/grasshopper-engine/build
      Start  1: digest-with-engine
 1/15 Test  #1: digest-with-engine ...............   Passed    0.07 sec
      Start  2: digest-with-provider
 2/15 Test  #2: digest-with-provider .............   Passed    0.06 sec
      Start  3: ciphers-with-engine
 3/15 Test  #3: ciphers-with-engine ..............   Passed    0.04 sec
      Start  4: ciphers-with-provider
 4/15 Test  #4: ciphers-with-provider ............   Passed    0.04 sec
      Start  5: curves
 5/15 Test  #5: curves ...........................   Passed    0.04 sec
      Start  6: parameters-with-engine
 6/15 Test  #6: parameters-with-engine ...........   Passed    0.03 sec
      Start  7: derive-with-engine
 7/15 Test  #7: derive-with-engine ...............   Passed    0.06 sec
      Start  8: sign/verify-with-engine
 8/15 Test  #8: sign/verify-with-engine ..........   Passed    0.12 sec
      Start  9: TLS-with-engine
 9/15 Test  #9: TLS-with-engine ..................   Passed    0.71 sec
      Start 10: context-with-engine
10/15 Test #10: context-with-engine ..............   Passed    0.02 sec
      Start 11: context-with-provider
11/15 Test #11: context-with-provider ............   Passed    0.01 sec
      Start 12: keyexpimp
12/15 Test #12: keyexpimp ........................   Passed    0.02 sec
      Start 13: gost89
13/15 Test #13: gost89 ...........................   Passed    0.02 sec
      Start 14: engine
14/15 Test #14: engine ...........................   Passed    3.56 sec
      Start 15: provider
15/15 Test #15: provider .........................***Failed    1.64 sec

93% tests passed, 1 tests failed out of 15

Total Test time (real) =   6.46 sec

The following tests FAILED:
     15 - provider (Failed)
Errors while running CTest
Output from these tests are in: /Users/ur20980/src/grasshopper-engine/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

And the complete log (including configuration and build): ossl3-build.txt

mouse07410 commented 3 years ago

Ping...?

beldmit commented 3 years ago

Will try to play with the new clang on Friday

beldmit commented 3 years ago

Using clang 12.0.1, I get segfault on digest-with-provider. Will try to investigate.

beldmit commented 3 years ago

Test output:

Test md_gost12_512: M1 from RFC 6986 (10.1.1) and GOST R 34.11-2012 (А.1.1): [MD 64] success
Test md_gost12_256: M1 from RFC 6986 (10.1.2) and GOST R 34.11-2012 (А.1.2): [MD 32] success
Test md_gost12_512: M2 from RFC 6986 (10.2.1) and GOST R 34.11-2012 (А.2.1): [MD 0] success
Test md_gost12_256: M2 from RFC 6986 (10.2.2) and GOST R 34.11-2012 (А.2.2): [MD 0] success
806BEB56E77F0000:error:0300009B:digital envelope routines:evp_mac_final:buffer too small:crypto/evp/mac_lib.c:156:
/home/dbelyavs/upstream/engine/test_digest.c:666: OpenSSL internal error: EVP_MAC_finalXOF(ctx, out, outsize)
mouse07410 commented 2 years ago

test-3-out.txt make-3-out.txt cmake-3-out.txt test-3-long-out.txt

beldmit commented 2 years ago

I have just patched the test that was failing on linux. Could you please retry?

mouse07410 commented 2 years ago

Here's with the OpenSSL-3.0.0 (released): cmake-3-out.txt make-3-out.txt test-3-out.txt test-3-long-out.txt

Here's with the OpenSSL-3.1.dev (master): test-3m-long-out.txt test-3m-out.txt make-3m-out.txt cmake-3m-out.txt

And I still think that renaming "grashopper" to "kuznyechik", especially with that spelling, was a bad idea. As they say, "ugly as a mule". ;-)

beldmit commented 2 years ago

I think I found the culprit:

https://github.com/gost-engine/engine/blob/8efda3214b5592a5267db96cb0d0da2a2c747a69/gost_prov_mac.c#L148

It should be EVP_DigestFinalXOF, but naïve replacement does not help.

beldmit commented 2 years ago

Could you please the corresponding fix? I still can't reproduce it but it should fix the issue.

index ab04a9c..1bad120 100644
--- a/gost_prov_mac.c
+++ b/gost_prov_mac.c
@@ -145,7 +145,7 @@ static int mac_final(void *mctx, unsigned char *out, size_t *outl,
         /* We ignore the error for GOST MDs that don't support setting
            the size */
         EVP_MD_CTX_ctrl(gctx->dctx, EVP_MD_CTRL_XOF_LEN, gctx->mac_size, NULL);
-        ret = EVP_DigestFinal_ex(gctx->dctx, out, &tmpoutl);
+        ret = EVP_DigestFinalXOF(gctx->dctx, out, gctx->mac_size);
     }
     if (outl != NULL)
         *outl = (size_t)gctx->mac_size;
mouse07410 commented 2 years ago

Funny. This change made things worse - now two tests fail: context-with-provider and provider.

ossl3m-build.txt ossl3-build.txt

Crash report doesn't help much:

Process:               test_context [9918]
Path:                  /Users/USER/*/test_context
Identifier:            test_context
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        ctest [9889]
Responsible:           Terminal [813]
User ID:               501

Date/Time:             2021-11-14 18:42:48.627 -0500
OS Version:            macOS 11.6.1 (20G224)
Report Version:        12
Bridge OS Version:     6.0 (19P548)
Anonymous UUID:        BD844EB9-9C6F-867E-78EB-1ACDA55970A0

Time Awake Since Boot: 380000 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
abort() called

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_kernel.dylib          0x00007fff204e592e __pthread_kill + 10
1   libsystem_pthread.dylib         0x00007fff205145bd pthread_kill + 263
2   libsystem_c.dylib               0x00007fff20469406 abort + 125
3   libcrypto.3.dylib               0x00000001077bdc3f OPENSSL_die + 26
4   test_context                    0x000000010768d226 main + 1670
5   libdyld.dylib                   0x00007fff2052ff3d start + 1
beldmit commented 2 years ago

Do I correctly assume that you've updated the code and applied the patch afterwards?

mouse07410 commented 2 years ago

Do I correctly assume that you've updated the code and applied the patch afterwards?

Yes. But I don't see any updates...? Latest commit from you that I pulled is 8efda3214b5592a5. And on the libprov submodule: 8a126e09547.

beldmit commented 2 years ago

Yes, it's a relevant state +patch from https://github.com/gost-engine/engine/issues/359#issuecomment-968343015

mouse07410 commented 2 years ago

Yes, that's the current status of the code on my machines. Plus, same error with both OpenSSL_3.0.0 and 3.1.dev.