Master fails oqs_tlssig test on MacOS

[mouse07410]

Describe the bug Current master fails oqs_tlssig test:

Successful build for source-based OpenSSL
Uri's DYLD_LIBRARY_PATH="/Users/ur20980/openssl-3/lib:/usr/local/lib:/opt/local/lib:"
Test setup:
LD_LIBRARY_PATH=/Users/ur20980/openssl-3/lib:/usr/local/lib:/opt/local/lib:
OPENSSL_APP=/Users/ur20980/openssl-3/bin/openssl
OPENSSL_CONF=/Users/ur20980/openssl-3/etc/openssl.cnf
OPENSSL_MODULES=/Users/ur20980/openssl-3/lib/ossl-modules
DYLD_LIBRARY_PATH=/Users/ur20980/openssl-3/lib:/usr/local/lib:/opt/local/lib:
No OQS-OpenSSL111 interop test because of absence of docker
Version information:
OpenSSL 3.2.0-dev  (Library: OpenSSL 3.2.0-dev )
Providers:
  base
    name: OpenSSL Base Provider
    version: 3.2.0
    status: active
    build info: 3.2.0-dev
    gettable provider parameters:
      name: pointer to a UTF8 encoded string (arbitrary size)
      version: pointer to a UTF8 encoded string (arbitrary size)
      buildinfo: pointer to a UTF8 encoded string (arbitrary size)
      status: integer (arbitrary size)
  default
    name: OpenSSL Default Provider
    version: 3.2.0
    status: active
    build info: 3.2.0-dev
    gettable provider parameters:
      name: pointer to a UTF8 encoded string (arbitrary size)
      version: pointer to a UTF8 encoded string (arbitrary size)
      buildinfo: pointer to a UTF8 encoded string (arbitrary size)
      status: integer (arbitrary size)
  oqs
    name: OpenSSL OQS Provider
    version: 0.5.0-dev
    status: active
    build info: OQS Provider v.0.5.0-dev (5dc4bbf) based on liboqs v.0.8.0-rc1
    gettable provider parameters:
      name: pointer to a UTF8 encoded string (arbitrary size)
      version: pointer to a UTF8 encoded string (arbitrary size)
      buildinfo: pointer to a UTF8 encoded string (arbitrary size)
      status: integer (arbitrary size)
  oqsprovider
    name: OpenSSL OQS Provider
    version: 0.5.0-dev
    status: active
    build info: OQS Provider v.0.5.0-dev (5dc4bbf) based on liboqs v.0.8.0-rc1
    gettable provider parameters:
      name: pointer to a UTF8 encoded string (arbitrary size)
      version: pointer to a UTF8 encoded string (arbitrary size)
      buildinfo: pointer to a UTF8 encoded string (arbitrary size)
      status: integer (arbitrary size)
  pkcs11
    name: PKCS#11 Provider
    version: 3.2.0
    status: active
    build info: 3.2.0-dev
    gettable provider parameters:
      name: pointer to a UTF8 encoded string (arbitrary size)
      version: pointer to a UTF8 encoded string (arbitrary size)
      buildinfo: pointer to a UTF8 encoded string (arbitrary size)
      status: integer (arbitrary size)
Cert gen/verify, CMS sign/verify, CA tests for all enabled OQS signature algorithms commencing: 

Test project /Users/ur20980/src/oqs-provider/_build
    Start 1: oqs_signatures
1/5 Test #1: oqs_signatures ...................   Passed    4.01 sec
    Start 2: oqs_kems
2/5 Test #2: oqs_kems .........................   Passed    0.43 sec
    Start 3: oqs_groups
3/5 Test #3: oqs_groups .......................   Passed    0.56 sec
    Start 4: oqs_tlssig
4/5 Test #4: oqs_tlssig .......................***Failed    0.24 sec
    Start 5: oqs_endecode
5/5 Test #5: oqs_endecode .....................   Passed    5.16 sec

80% tests passed, 1 tests failed out of 5

Total Test time (real) =  10.41 sec

The following tests FAILED:
      4 - oqs_tlssig (Failed)

To Reproduce Steps to reproduce the behavior:

1. Run and observe. ;-)

Expected behavior All tests passing, like:

Test project /Users/ur20980/src/oqs-provider/_build
    Start 1: oqs_signatures
1/5 Test #1: oqs_signatures ...................   Passed    3.99 sec
    Start 2: oqs_kems
2/5 Test #2: oqs_kems .........................   Passed    0.45 sec
    Start 3: oqs_groups
3/5 Test #3: oqs_groups .......................   Passed    0.55 sec
    Start 4: oqs_tlssig
4/5 Test #4: oqs_tlssig .......................   Passed    0.15 sec
    Start 5: oqs_endecode
5/5 Test #5: oqs_endecode .....................   Passed    5.74 sec

100% tests passed, 0 tests failed out of 5

Logs

Build that is failing tests (OpenSSL-3.2.0-dev): tests-out-s.txt build-out-s.txt

LastTest.log

Build that passes the tests (OpenSSL-3.1.0): tests-out.txt build-out.txt

Environment (please complete the following information):

• Platform: Apple computers, both Intel x86_64 and M2 Apple Silicon
• OS: MacOS Ventura 13.4
• OpenSSL version: 3.2.0-dev (failing)

[mouse07410]

In case it helps:

$ openssl3 list -signature-algorithms -provider oqs
  { 1.2.840.113549.1.1.1, 2.5.8.1.1, RSA, rsaEncryption } @ default
  HMAC @ default
  SIPHASH @ default
  POLY1305 @ default
  CMAC @ default
  { 1.2.840.10040.4.1, 1.2.840.10040.4.3, 1.3.14.3.2.12, 1.3.14.3.2.13, 1.3.14.3.2.27, DSA, DSA-old, DSA-SHA, DSA-SHA1, DSA-SHA1-old, dsaEncryption, dsaEncryption-old, dsaWithSHA, dsaWithSHA1, dsaWithSHA1-old } @ default
  { 1.3.101.112, ED25519 } @ default
  { 1.3.101.113, ED448 } @ default
  { 1.2.156.10197.1.301, SM2 } @ default
  ECDSA @ default
  rsa3072_falcon512 @ oqs
  falcon1024 @ oqs
  p521_falcon1024 @ oqs
  sphincssha2128fsimple @ oqs
  p256_sphincssha2128fsimple @ oqs
  rsa3072_sphincssha2128fsimple @ oqs
  sphincssha2128ssimple @ oqs
  p256_sphincssha2128ssimple @ oqs
  rsa3072_sphincssha2128ssimple @ oqs
  sphincssha2192fsimple @ oqs
  p384_sphincssha2192fsimple @ oqs
  sphincsshake128fsimple @ oqs
  p256_sphincsshake128fsimple @ oqs
  rsa3072_sphincsshake128fsimple @ oqs
  dilithium2 @ oqs
  p256_dilithium2 @ oqs
  rsa3072_dilithium2 @ oqs
  dilithium3 @ oqs
  p384_dilithium3 @ oqs
  dilithium5 @ oqs
  p521_dilithium5 @ oqs
  falcon512 @ oqs
  p256_falcon512 @ oqs
$ 

[baentsch]

What would help is output of the failure itself (running ctest/runtests.sh with option -V).

[mouse07410]

What would help is output of the failure itself (running ctest/runtests.sh with option -V).

Could you please give the exact command as I would type it?

[baentsch]

./scripts/runtests.sh -V

[mouse07410]

LastTest.log build-out-s.txt tests-out-s.txt

[baentsch]

WOW -- I've never seen so much log output. What software generates this??? The error cause is obvious, though: The test certs are not found, e.g.

calling fopen(/Users/ur20980/src/oqs-provider/test/../tmp/p384_dilithium3_srv.crt

I did change something to make this more robust in https://github.com/open-quantum-safe/oqs-provider/pull/175: Would you mind running that branch (instead)? Alternatively/in addition, could you please check whether the cert files did get generated (somewhere else)?

[mouse07410]

WOW -- I've never seen so much log output. What software generates this???

Hmm... Yours, probably? ;-)

in addition, could you please check whether the cert files did get generated (somewhere else)?

Looks like they were either deleted, or not generated at all:

$ pwd
/Users/ur20980/src/oqs-provider
$ find . -name '*.crt' -print
$
$ ll tmp
total 0
drwxr-xr-x   2 ur20980  staff   64 May 30 22:48 ./
drwxr-xr-x  23 ur20980  staff  736 Jun  1 10:44 ../
$ 

I did change something to make this more robust in https://github.com/open-quantum-safe/oqs-provider/pull/175: Would you mind running that branch (instead)?

You mean - pull that PR and try it? Sure. Still failing. BTW, what was the reason for changing oqsprovider.0.5.0-dev.dylib to oqsprovider.0.5.0.dylib? I wouldn't mind being told of such changes, rather than discover them from messages like

cp: _build/lib/oqsprovider.0.5.0-dev.dylib: No such file or directory

tests-out-s.txt build-out-s.txt

LastTest.log

Update

Since other tests appear to succeed, something probably did generate certificates? If so, what deleted them? Why is tmp/ empty?

[baentsch]

Hmm... Yours, probably? ;-)

I've never seen that and don't recall coding that. For the fun of it, switch off pkcs11 and see whether there's less output...

Looks like they were either deleted, or not generated at all:

Then the latter: When "-V" has been passed, there should be a line "Testing " following the "...commencing" statement. But there's nothing. This in turn puzzles me: This is only possible if "openssl list -signature-algorithms | grep oqsprovider" didn't return any oqs algorithm -- and I can't think of a reason for this after "openssl list -providers" showed "oqsprovider" properly registered (as is the case).

Since other tests appear to succeed,

not all tests depend on proper certs to be there. Test 4 needs them, though. But their generation as per the above failed for some reason....

[mouse07410]

For the fun of it, switch off pkcs11 and see whether there's less output...

Nope, exactly the same ballpark (about 9.8MB).

I can't think of a reason for this after "openssl list -providers" showed "oqsprovider" properly registered (as is the case).

I probably can. The provider is named oqs, not `oqsprovider:

$ openssl3 list -signature-algorithms | grep oqsprovider
$ openssl3 list -signature-algorithms | grep oqs        
  rsa3072_falcon512 @ oqs
  falcon1024 @ oqs
  p521_falcon1024 @ oqs
  sphincssha2128fsimple @ oqs
  p256_sphincssha2128fsimple @ oqs
  rsa3072_sphincssha2128fsimple @ oqs
  sphincssha2128ssimple @ oqs
  p256_sphincssha2128ssimple @ oqs
  rsa3072_sphincssha2128ssimple @ oqs
  sphincssha2192fsimple @ oqs
  p384_sphincssha2192fsimple @ oqs
  sphincsshake128fsimple @ oqs
  p256_sphincsshake128fsimple @ oqs
  rsa3072_sphincsshake128fsimple @ oqs
  dilithium2 @ oqs
  p256_dilithium2 @ oqs
  rsa3072_dilithium2 @ oqs
  dilithium3 @ oqs
  p384_dilithium3 @ oqs
  dilithium5 @ oqs
  p521_dilithium5 @ oqs
  falcon512 @ oqs
  p256_falcon512 @ oqs
$ 

[mouse07410]

1. Does "-V" really give you something useful here? Perhaps, we should drop it now?
2. Could you add printing out the algorithms from within your test-scripts?
3. After relevant certificates are generated (assuming they are), could you add something like ls -alF tmp/ or wherever they are placed?

Update

What's the role of Docker in oqs_tlssig test? It looks like (some?) certs for TLS are generated only inside a docker container?

Update 2

I don't see any of the "echo" that normal scripts lilke oqsprovider-certgen.sh are supposed to output. Are those scripts even invoked? Is there any way to avoid suppressing that output? BTW, I've added ls -alF tmp/ to oqsprovider-certgen.sh, and that output does not show up. So, either it's redirected to /dev/null somewhere, or that script was not invoked at all.

[baentsch]

I probably can. The provider is named oqs, not `oqsprovider:

This doesn't make sense: Two instances, "oqs" and "oqsprovider" are listed in your output of "list -providers": How could only one support OQS signature algorithms but not the other? Now added a sanity check for this.

Does "-V" really give you something useful here? Perhaps, we should drop it now?

Yes and No in that order: This is activating verbose ctest output. Without this, error messages are suppressed and we wouldn't know that the certs have not been generated.

Could you add printing out the algorithms from within your test-scripts?

This already happens: https://github.com/open-quantum-safe/oqs-provider/blob/407221019706b104318a0ed9baf6185fb295b83a/scripts/runtests.sh#L153 (if "-V" is set).

(assuming they are)

They are not. Output thus wouldn't happen. I now added a sanity check, stopping testing when no OQS sig algs have been found. Directory is now output if certs have been generated. See https://github.com/open-quantum-safe/oqs-provider/pull/175/commits/e6f4825dd8540e24ccf7f3be699157f72efac286.

I don't see any of the "echo" that normal scripts lilke oqsprovider-certgen.sh are supposed to output.

This is all redirected to "interop.log" (as otherwise too much logging output is generated by openssl itself). This in turn is output in case an error occurs: https://github.com/open-quantum-safe/oqs-provider/blob/07107deb2a9dc7d94ee7e52e7b34e1eb32a42a89/scripts/runtests.sh#L21

[mouse07410]

This is all redirected to "interop.log" (as otherwise too much logging output is generated by openssl itself).

Doch... I completely forgot about "interop.log".

But... it does not seem to be generated anymore! I don't have it:

The following tests FAILED:
      4 - oqs_tlssig (Failed)
Errors while running CTest
Output from these tests are in: /Users/ur20980/src/oqs-provider/_build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.

Tests failed.
$ ll
total 21024
drwxr-xr-x   23 ur20980  staff      736 Jun  2 12:20 ./
drwxr-xr-x  103 ur20980  staff     3296 Jun  2 12:19 ../
drwxr-xr-x    3 ur20980  staff       96 Jun  1 08:40 .circleci/
drwxr-xr-x   15 ur20980  staff      480 Jun  1 11:34 .git/
drwxr-xr-x    6 ur20980  staff      192 Apr  2 09:17 .github/
-rw-r--r--    1 ur20980  staff      270 May 24 22:38 .gitignore
-rw-r--r--    1 ur20980  staff    13000 May 19 20:53 ALGORITHMS.md
-rw-r--r--@   1 ur20980  staff   108029 Apr  8 23:12 ALGORITHMS.pdf
-rw-r--r--    1 ur20980  staff    30813 Apr  8 23:14 ALGORITHMS.tex
-rw-r--r--    1 ur20980  staff     1378 Jun  1 11:34 CMakeLists.txt
-rw-r--r--    1 ur20980  staff     1156 Apr  2 09:17 LICENSE.txt
-rw-r--r--    1 ur20980  staff    19998 May 19 20:53 README.md
-rw-r--r--    1 ur20980  staff     5074 Jun  1 11:34 RELEASE.md
drwxr-xr-x   13 ur20980  staff      416 Jun  2 12:20 _build/
-rw-r--r--    1 ur20980  staff    33530 Jun  2 12:20 build-out-s.txt
-rw-r--r--    1 ur20980  staff    32229 Jun  1 11:49 build-out.txt
drwxr-xr-x   17 ur20980  staff      544 May 30 22:48 oqs-template/
drwxr-xr-x   18 ur20980  staff      576 Jun  1 08:40 oqsprov/
drwxr-xr-x   16 ur20980  staff      512 Jun  1 14:09 scripts/
drwxr-xr-x   16 ur20980  staff      512 Jun  1 14:09 test/
-rw-r--r--    1 ur20980  staff  9896723 Jun  2 12:20 tests-out-s.txt
-rw-r--r--    1 ur20980  staff     2643 Jun  1 11:49 tests-out.txt
drwxr-xr-x    2 ur20980  staff       64 May 30 22:48 tmp/
$ fd -u interop.log 
$ fd -u interop    
$ 

LastTest.log build-out-s.txt tests-out-s.txt

[mouse07410]

Still persists.

[baentsch]

The key issue is that openssl list -signature-algorithms doesn't seem to return PQ sig algs in oqsprovider. But I don't understand why this code isn't triggered (at least no output appears in your logs): Did you run with -V? https://github.com/open-quantum-safe/oqs-provider/blob/9c2a750d154b400ce79e500a01056e399294c65c/scripts/runtests.sh#L169-L176

[mouse07410]

But I don't understand why this code isn't triggered (at least no output appears in your logs):

Neither do I. :-(

Did you run with -V?

Yes. But judge for yourself:

tests-out-s.txt build-out-s.txt

I would very much prefer if there was a way to tell the script to (a) stop wiping out the certs it created when it thinks they aren't needed anymore, and (b) list the content of the tmp/ and maybe other relevant directories, so we can better guess what happened based on the files present, rather than merely on the console output.

[baentsch]

I would very much prefer if there was a way to tell the script to (a) stop wiping out the certs it created when it thinks they aren't needed anymore, and (b) list the content of the tmp/ and maybe other relevant directories, so we can better guess what happened based on the files present, rather than merely on the console output.

This is exactly what should happen. There's no code deleting interop.log or tmp dir. Also, there's output missing in your "tests-out-s.txt" between lines 51 and 53 (exactly that showing the cert-gen & testing of all sig algs). Here's the relevant code that shows what should happen (on branch mb-preprel): https://github.com/open-quantum-safe/oqs-provider/blob/9c2a750d154b400ce79e500a01056e399294c65c/scripts/runtests.sh#L159-L176

So can you please provide the output of git status && git diff (and/or verify that those lines are present unchanged in your copy of "script/runtests.sh")? Please also feel free to add the instruction "$OPENSSL_APP list -signature-algorithms" right above the line "# Run interop-tests:": I have the gut feeling it will return no output in your setup. I do not understand, though, how it can proceed to execute the remaining tests as the (then unset) variable certsgenerated should prohibit this.

[mouse07410]

$ git status
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    build-out-s.txt
    build-out.txt
    tests-out-s.txt
    tests-out.txt

nothing added to commit but untracked files present (use "git add" to track)
$ git diff
$ pwd
/Users/ur20980/src/oqs-provider
$ git remote -v
origin  https://github.com/open-quantum-safe/oqs-provider.git (fetch)
origin  https://github.com/open-quantum-safe/oqs-provider.git (push)
$ 

[mouse07410]

OK. more work - more info.

First, I polished your runtests.sh script a bit, to make sure it does the right things:

diff --git a/scripts/runtests.sh b/scripts/runtests.sh
index b9975e4..dfc06b7 100755
--- a/scripts/runtests.sh
+++ b/scripts/runtests.sh
@@ -113,7 +113,7 @@ echo "OPENSSL_APP=$OPENSSL_APP"
 echo "OPENSSL_CONF=$OPENSSL_CONF"
 echo "OPENSSL_MODULES=$OPENSSL_MODULES"
 if [[ "$OSTYPE" == "darwin"* ]]; then
-echo "DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH"
+   echo "DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH"
 fi

 # check if we can use docker or not:
@@ -130,21 +130,23 @@ fi
 export LOCALTESTONLY="Yes"

 echo "Version information:"
-$OPENSSL_APP version
+${OPENSSL_APP} version

 # Disable testing for version 3.0.1: Buggy as hell:
-$OPENSSL_APP version | grep "OpenSSL 3.0.1" > /dev/null
+${OPENSSL_APP} version | grep "OpenSSL 3.0.1" > /dev/null
 if [ $? -eq 0 ]; then
    echo "Skipping testing of buggy OpenSSL 3.0.1"
    exit 0
 fi

-$OPENSSL_APP list -providers -verbose -provider-path _build/lib -provider oqsprovider
+${OPENSSL_APP} list -providers -verbose -provider-path _build/lib -provider oqsprovider
 if [ $? -ne 0 ]; then
    echo "Baseline openssl invocation failed. Exiting test."
    exit 1
 fi

+${OPENSSL_APP} list -signature-algorithms | tee "${PWD}/sig-algs.txt"
+
 # Run interop-tests:
 echo "Cert gen/verify, CMS sign/verify, CA tests for all enabled OQS signature algorithms commencing: "
 for alg in `$OPENSSL_APP list -signature-algorithms | grep oqsprovider | sed -e "s/ @ oqsprovider//g" | sed -e "s/^  //g"`

Here are the logs: tests-out-s.txt sig-algs.txt build-out-s.txt

Now, if you see at the line 241 and following of tests-out-s.txt, you'll see that it did not (for whatever reason) pick your oqs.cnf, but used the system-wide openssl.cnf from the $OPENSSL_ROOT_DIR. And that config, naturally, has no oqsprovider, only oqs provider.

So, the problem seems to be - OPENSSL_CONF is messed up for that test.

Also, note that other tests/scripts that apparently exercise OQS signatures, seem to work just fine.

Update

Could you please prevent lines like the following from being printed/logged?

: Name: ADH-CAMELLIA256-SHA256:
4: Algo = 00000002/00000004/00000200/00000010/00000303 Algo_strength = 00000028
4: Action = 4
4:
4: Name: DHE-RSA-CAMELLIA256-SHA256:
4: Algo = 00000002/00000001/00000200/00000010/00000303 Algo_strength = 00000028
4:
4: Name: DHE-DSS-CAMELLIA256-SHA256:
4: Algo = 00000002/00000002/00000200/00000010/00000303 Algo_strength = 00000028
4:
4: Name: CAMELLIA256-SHA256:
4: Algo = 00000001/00000001/00000200/00000010/00000303 Algo_strength = 00000028
4:
4: Name: ADH-CAMELLIA128-SHA256:
4: Algo = 00000002/00000004/00000100/00000010/00000303 Algo_strength = 00000028
4: Action = 4
4:
4: Name: DHE-RSA-CAMELLIA128-SHA256:
4: Algo = 00000002/00000001/00000100/00000010/00000303 Algo_strength = 00000028
4:
4: Name: DHE-DSS-CAMELLIA128-SHA256:
4: Algo = 00000002/00000002/00000100/00000010/00000303 Algo_strength = 00000028
4:
4: Name: CAMELLIA128-SHA256:
4: Algo = 00000001/00000001/00000100/00000010/00000303 Algo_strength = 00000028

They are the bulk of the useless (for me, at least) noise that interferes with going through the output and analyzing it.

[mouse07410]

I don't understand.

When I run the following script, all the tests pass:

#!/bin/bash

# Cleaning up previous builds
make clean
#rm -rf tmp/*
rm -f interop.log #interop-3.log

# Set env var - flags
OQSPROV=1
OQSKM=1
OQSKEY=1

unset OPENSSL_INSTALL

set DYLD="${DYLD_LIBRARY_PATH}"

# Build for local sources of master branch of OpenSSL-3.2+
if [ -d $HOME/openssl-3 ]; then
    rm -rf _build

    export DYLD_LIBRARY_PATH="${HOME}/openssl-3/lib:/usr/local/lib:/opt/local/lib:${DYLD}"
    export LD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}"

    OPENSSL_ROOT_DIR="$HOME/openssl-3"
    OPENSSL_DIR="$OPENSSL_ROOT_DIR"
    OPENSSL_INSTALL="$OPENSSL_DIR"
    OPENSSL_APP="$OPENSSL_ROOT_DIR/bin/openssl"
    OPENSSL="$OPENSSL_APP"
    OPENSSL_LIB_DIR="$OPENSSL_ROOT_DIR/lib"
    OPENSSL_INCLUDE_DIR="$OPENSSL_ROOT_DIR/include"

    #OPENSSL_MODULES="$OPENSSL_ROOT_DIR/lib/ossl-modules"
    OPENSSL_MODULES="${PWD}/_build/lib"
        #unset OPENSSL_MODULES

    #OPENSSL_CONF="$OPENSSL_ROOT_DIR/etc/openssl.cnf"
    OPENSSL_CONF="${PWD}/tests/oqs.cnf"
        #unset OPENSSL_CONF

    echo "Building for source-based OpenSSL-3.2.x-dev..."
    env | grep OPENSSL > build-out-s.txt
    echo "" >> build-out-s.txt
    cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_ENCODING_LIB=OFF -DOPENSSL_ROOT_DIR="$HOME/src/openssl" -DCMAKE_C_FLAGS="$CFLAGS -g -I${OPENSSL_INCLUDE_DIR} -L${OPENSSL_LIB_DIR} " -DCMAKE_VERBOSE_MAKEFILE:BOOL=True -S . -B _build 2>&1 | tee -a build-out-s.txt
    cmake --build _build 2>&1 | tee -a build-out-s.txt
    if [ -x _build/lib/oqsprovider.dylib ]; then
        echo "Successful build for source-based OpenSSL"
        echo "Uri's DYLD_LIBRARY_PATH=\"${DYLD_LIBRARY_PATH}\""
        scripts/runtests.sh -V 2>&1 | tee tests-out-s.txt
        #cp _build/lib/oqsprovider.0.5.0-dev.dylib "$OPENSSL_MODULES"
    else
        echo "Apparently, building for source-based OpenSSL-3.2.x-dev failed"
        echo ""
    fi
        mv "interop.log" "interop-3.log"
else
    echo ""
    echo "Sources of OpenSSL-3.2.x-dev not found, skipping..."
    echo ""
fi

exit

build-out-s.txt tests-out-s.txt sig-algs.txt

However, you can see that sig-algs.txt shows no OQS algorithms at all!

But when I try to check for them manually:

$ pwd
/Users/ur20980/src/oqs-provider
$ OPENSSL_MODULES=_build/lib OPENSSL_CONF=test/oqs.cnf openssl list -signature-algorithms | grep oqs | sed -e "s/ @ oqs.*//g"
  p384_dilithium3
  p521_dilithium5
  falcon512
  p256_falcon512
  rsa3072_falcon512
  falcon1024
  p521_falcon1024
  sphincssha2128fsimple
  p256_sphincssha2128fsimple
  rsa3072_sphincssha2128fsimple
  sphincssha2128ssimple
  p256_sphincssha2128ssimple
  rsa3072_sphincssha2128ssimple
  sphincssha2192fsimple
  p384_sphincssha2192fsimple
  sphincsshake128fsimple
  p256_sphincsshake128fsimple
  rsa3072_sphincsshake128fsimple
  dilithium2
  p256_dilithium2
  rsa3072_dilithium2
  dilithium3
  dilithium5
$ 

---

Here's my modified version of runtests.sh:

diff --git a/scripts/runtests.sh b/scripts/runtests.sh
index b9975e4..525f0bc 100755
--- a/scripts/runtests.sh
+++ b/scripts/runtests.sh
@@ -113,7 +113,7 @@ echo "OPENSSL_APP=$OPENSSL_APP"
 echo "OPENSSL_CONF=$OPENSSL_CONF"
 echo "OPENSSL_MODULES=$OPENSSL_MODULES"
 if [[ "$OSTYPE" == "darwin"* ]]; then
-echo "DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH"
+   echo "DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH"
 fi

 # check if we can use docker or not:
@@ -130,24 +130,29 @@ fi
 export LOCALTESTONLY="Yes"

 echo "Version information:"
-$OPENSSL_APP version
+${OPENSSL_APP} version

 # Disable testing for version 3.0.1: Buggy as hell:
-$OPENSSL_APP version | grep "OpenSSL 3.0.1" > /dev/null
+${OPENSSL_APP} version | grep "OpenSSL 3.0.1" > /dev/null
 if [ $? -eq 0 ]; then
    echo "Skipping testing of buggy OpenSSL 3.0.1"
    exit 0
 fi

-$OPENSSL_APP list -providers -verbose -provider-path _build/lib -provider oqsprovider
+${OPENSSL_APP} list -providers -verbose -provider-path _build/lib -provider oqsprovider
 if [ $? -ne 0 ]; then
    echo "Baseline openssl invocation failed. Exiting test."
    exit 1
 fi

+echo ""
+echo "Known signature algorithms:"
+${OPENSSL_APP} list -signature-algorithms | tee "${PWD}/sig-algs.txt"
+echo ""
+
 # Run interop-tests:
 echo "Cert gen/verify, CMS sign/verify, CA tests for all enabled OQS signature algorithms commencing: "
-for alg in `$OPENSSL_APP list -signature-algorithms | grep oqsprovider | sed -e "s/ @ oqsprovider//g" | sed -e "s/^  //g"`
+for alg in `$OPENSSL_APP list -signature-algorithms | grep oqs | sed -e "s/ @ oqs.*//g" | sed -e "s/^  //g"`
 do 
    if [ "$1" = "-V" ]; then
       echo "Testing $alg"

[mouse07410]

Apparently, ${OPENSSL_APP} list -signature-algorithms fails to list any from the oqsprovider.

However, when explicitly specified like ${OPENSSL_APP} list -signature-algorithms -provider-path _build/lib -provider oqsprovider, it lists all of them appropriately.

Update

My latest. I'm almost ready to give up.

#!/bin/bash

# Cleaning up previous builds
rm -rf _build
make clean
rm -rf tmp/*
rm -f interop.log #interop-3.log

# Set env var - flags
OQSPROV=1
OQSKM=1
OQSKEY=1

unset OPENSSL_INSTALL

set DYLD="${DYLD_LIBRARY_PATH}"

# Build for local sources of master branch of OpenSSL-3.2+
if [ -d $HOME/openssl-3 ]; then
    rm -rf _build

    export DYLD_LIBRARY_PATH="${HOME}/openssl-3/lib:/usr/local/lib:/opt/local/lib:${DYLD}"
    export LD_LIBRARY_PATH="${DYLD_LIBRARY_PATH}"

    OPENSSL_ROOT_DIR="$HOME/openssl-3"
    OPENSSL_DIR="$OPENSSL_ROOT_DIR"
    OPENSSL_INSTALL="$OPENSSL_DIR"
    OPENSSL_APP="$OPENSSL_ROOT_DIR/bin/openssl"
    OPENSSL="$OPENSSL_APP"
    OPENSSL_LIB_DIR="$OPENSSL_ROOT_DIR/lib"
    OPENSSL_INCLUDE_DIR="$OPENSSL_ROOT_DIR/include"

    #OPENSSL_MODULES="$OPENSSL_ROOT_DIR/lib/ossl-modules"
    OPENSSL_MODULES="${PWD}/_build/lib"
    #unset OPENSSL_MODULES

    #OPENSSL_CONF="$OPENSSL_ROOT_DIR/etc/openssl.cnf"
    OPENSSL_CONF="${PWD}/tests/oqs.cnf"
    #unset OPENSSL_CONF

    echo "Building for source-based OpenSSL-3.2.x-dev..."
    env | grep OPENSSL > build-out-s.txt
    echo "" >> build-out-s.txt
    cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_ENCODING_LIB=OFF -DOPENSSL_ROOT_DIR="$HOME/src/openssl" -DCMAKE_C_FLAGS="$CFLAGS -g -I${OPENSSL_INCLUDE_DIR} -L${OPENSSL_LIB_DIR} " -DCMAKE_VERBOSE_MAKEFILE:BOOL=True -S . -B _build 2>&1 | tee -a build-out-s.txt
    cmake --build _build 2>&1 | tee -a build-out-s.txt
    if [ -x _build/lib/oqsprovider.dylib ]; then
        echo "Successful build for source-based OpenSSL"
        echo "Uri's DYLD_LIBRARY_PATH=\"${DYLD_LIBRARY_PATH}\""
        scripts/runtests.sh -V 2>&1 | tee tests-out-s.txt
        #cp _build/lib/oqsprovider.0.5.0-dev.dylib "$OPENSSL_MODULES"
    else
        echo "Apparently, building for source-based OpenSSL-3.2.x-dev failed"
        echo ""
    fi
else
    echo ""
    echo "Sources of OpenSSL-3.2.x-dev not found, skipping..."
    echo ""
fi

exit

diff --git a/scripts/runtests.sh b/scripts/runtests.sh
index b9975e4..c09fbe4 100755
--- a/scripts/runtests.sh
+++ b/scripts/runtests.sh
@@ -113,7 +113,7 @@ echo "OPENSSL_APP=$OPENSSL_APP"
 echo "OPENSSL_CONF=$OPENSSL_CONF"
 echo "OPENSSL_MODULES=$OPENSSL_MODULES"
 if [[ "$OSTYPE" == "darwin"* ]]; then
-echo "DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH"
+   echo "DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH"
 fi

 # check if we can use docker or not:
@@ -130,24 +130,32 @@ fi
 export LOCALTESTONLY="Yes"

 echo "Version information:"
-$OPENSSL_APP version
+${OPENSSL_APP} version

 # Disable testing for version 3.0.1: Buggy as hell:
-$OPENSSL_APP version | grep "OpenSSL 3.0.1" > /dev/null
+${OPENSSL_APP} version | grep "OpenSSL 3.0.1" > /dev/null
 if [ $? -eq 0 ]; then
    echo "Skipping testing of buggy OpenSSL 3.0.1"
    exit 0
 fi

-$OPENSSL_APP list -providers -verbose -provider-path _build/lib -provider oqsprovider
+${OPENSSL_APP} list -providers -verbose -provider-path _build/lib -provider oqsprovider
 if [ $? -ne 0 ]; then
    echo "Baseline openssl invocation failed. Exiting test."
    exit 1
 fi

+echo ""
+echo "Known providers:"
+${OPENSSL_APP} list -providers -provider-path _build/lib -provider oqsprovider | tee "${PWD}/sig-algs.txt"
+echo "" >> "${PWD}/sig-algs.txt" | tee -a "${PWD}/sig-algs.txt"
+echo "Known signature algorithms:" | tee -a "${PWD}/sig-algs.txt"
+${OPENSSL_APP} list -signature-algorithms -provider-path _build/lib -provider oqsprovider | tee -a "${PWD}/sig-algs.txt"
+echo ""
+
 # Run interop-tests:
 echo "Cert gen/verify, CMS sign/verify, CA tests for all enabled OQS signature algorithms commencing: "
-for alg in `$OPENSSL_APP list -signature-algorithms | grep oqsprovider | sed -e "s/ @ oqsprovider//g" | sed -e "s/^  //g"`
+for alg in `$OPENSSL_APP list -signature-algorithms -provider-path "_build/lib" | grep oqs | sed -e "s/ @ oqs.*//g" | sed -e "s/^  //g"`
 do 
    if [ "$1" = "-V" ]; then
       echo "Testing $alg"

tests-out-s.txt sig-algs.txt build-out-s.txt

[baentsch]

Well, these scripts explain pretty much everything: 1) The mysterious vanishing of logfiles seems to be caused by your "wrapper script": At the start, interop.log is deleted and then not re-generated as no signature algorithm is found 2) The unsuccessful detection of sig algs in the latest run is caused by a missing "-provider oqsprovider" statement in the for loop, explaining the difference between "list" and execution. 3) Most importantly, you have set the wrong openssl.cnf file for the certgen test: test/oqs.cnf is only meant as an example for provider-activation. Only when using scripts/openssl-ca.cnf certificates can be correctly created, so please either fix that in your wrapper (or remove the explicit setting of the env var OPENSSL_CONF as runtests.sh will set it correctly if not externally set).

What I still don't understand, though, is why the logfile shows pkcs11 provider as being operational (but not shown at start of test): (At least the default) openssl cnf file (test/oqs.cnf) does not contain it. So I presume that file contains further changes I don't know about -- and that surely have a reading on the operation of the test: See above: If you didn't also add X509 structures to it, certgen will fail (even if it gets started).

I'm almost ready to give up.

Me too: Too many moving pieces (config files) :-(

[mouse07410]

At the start, interop.log is deleted and then not re-generated as no signature algorithm is found

I see.

At the start, interop.log is deleted

But of course! I'm only interested in the log produced by the current run, not the one left behind from whatever succeeded or failed before it.

The unsuccessful detection of sig algs in the latest run is caused by a missing "-provider oqsprovider" statement in the for loop, explaining the difference between "list" and execution.

But that was your script - I did not touch that part: https://github.com/open-quantum-safe/oqs-provider/blob/07107deb2a9dc7d94ee7e52e7b34e1eb32a42a89/scripts/runtests.sh#L150

Most importantly, you have set the wrong openssl.cnf file for the certgen test: test/oqs.cnf is only meant as an example for provider-activation.

I see. I did not realize this - and started experimenting with OPENSSL_CONF after I noticed that the tests used the system-wide openssl.cnf with providers installed there, instead of the just-buit and being-tested oqsprovider. Hence the oqs (installed system-wide) vs. oqsprovider (what your package builds and tests).

Only when using scripts/openssl-ca.cnf certificates can be correctly created, so please either fix that in your wrapper (or remove the explicit setting of the env var OPENSSL_CONF as runtests.sh will set it correctly if not externally set).

One reason I was explicitly setting OPENSSL_CONF was that my ~/.zprofile sets it so that the "normal" commands and apps use the correct OpenSSL config. For OpenSSL-3.2.0-dev, naturally, I have to point it to a different config file...

I think when my über-script did unset OPENSSL_CONF, there were problems. But now, with unset OPENSSL_CONF all tests pass for OpenSSL-3.2.0-dev.

BTW, what about OPENSSL_MODULES env var? I've set it to ${PWD}/_build/lib.

So I presume that file contains further changes I don't know about . . .

Nope, it does not. That was a complete file.

[baentsch]

But that was your script - I did not touch that part:

Well, your script shows this difference:

+for alg in $OPENSSL_APP list -signature-algorithms -provider-path "_build/lib" | grep oqs | sed -e "s/ @ oqs.*//g" | sed -e "s/^ //g"

This adds "-provider-path" but not "-provider". Therefore the logic you used to print before

+${OPENSSL_APP} list -signature-algorithms -provider-path _build/lib -provider oqsprovider | tee -a "${PWD}/sig-algs.txt"

(that circumvented the use of the OPENSSL_CONF file) is OK but the run logic above (that did not set "-provider" and thus relies on the file in OPENSSL_CONF) is not (working).

But now, with unset OPENSSL_CONF all tests pass for OpenSSL-3.2.0-dev.

Hurrah!

BTW, what about OPENSSL_MODULES env var? I've set it to ${PWD}/_build/lib.

That is OK (and would be set so by "runtests.sh" identically).

Nope, it does not. That was a complete file.

But I don't see it (i.e., your iteration of "oqs.cnf") shown above anywhere. All those many log entries are caused by pkcs11. Just look at your latest "tests-out-s.txt" file above:

4: Configuring provider pkcs11
4: Provider command: module = /Users/ur20980/openssl-3/lib/ossl-modules/pkcs11.dylib
4: Provider command: pkcs11-module-quirks = no-deinit
4: Provider command: pkcs11-module-login-behavior = auto
4: Provider command: pkcs11-module-cache-pins = cache
4: Provider command: pkcs11-module-path = /opt/p11kit/lib/p11-kit-proxy.dylib

But in sum, the key statement is that tests pass on OSX using #175, right? I think I'll then proceed and merge that too as the liboqs 0.8.0 release just was finalized so you can give that new "main" a new look before we release 0.5.0 of oqsprovider.

[mouse07410]

But in sum, the key statement is that tests pass on OSX using https://github.com/open-quantum-safe/oqs-provider/pull/175, right?

Right.

I think I'll then proceed and merge that too as the liboqs 0.8.0 release just was finalized so you can give that new "main" a new look before we release 0.5.0 of oqsprovider.

Sounds good to me. Thanks!