search

latchset / pkcs11-provider

A pkcs#11 provider for OpenSSL 3.0+
Other
64 stars 39 forks source link

On MacOS provider fails all tests but one #179

Closed mouse07410 closed 1 year ago

mouse07410 commented 1 year ago

Describe the bug All tests fail, except for tls_softtokn.

It appears that the tests omit to load this provider?

To Reproduce Steps to reproduce the behavior:

  1. Clone, apply the patch to make compilation possible, configure, build.
  2. Do make check
  3. See error (8 tests skipped, 9 tests failed, 1 test passed).

test-suite.log tls-softhsm-proxy.log tls-softokn.log readkeys-softhsm-proxy.log readkeys-softokn.log session-softhsm-proxy.log session-softokn.log genkey-softhsm.log genkey-softokn.log digests-softhsm-proxy.log digests-softokn.log rsapss-softokn.log hkdf-softokn.log oaepsha2-softokn.log eccsha2-softokn.log certs-softhsm-proxy.log certs-softokn.log basic-softhsm-proxy.log basic-softokn.log setup-softhsm.log setup-softokn.log

Expected behavior All tests passed.

Operating environment (please complete the following information):

Token and application used (please complete the following information):

Additional context Using OpenSSL master (3.2.0dev).

Here's the patch to compile pkcs11-provider - without it (at least on MacOS) it fails to get ssize_t type and load definition for snprintf() function.

diff --git a/src/provider.h b/src/provider.h
index 07b66c8..23a9e35 100644
--- a/src/provider.h
+++ b/src/provider.h
@@ -10,6 +10,7 @@
 #include <stdbool.h>

 #include "pkcs11.h"
+#include <sys/types.h>
 #include <openssl/core_dispatch.h>
 #include <openssl/core_object.h>
 #include <openssl/types.h>
diff --git a/src/session.c b/src/session.c
index 57e4169..0e1339e 100644
--- a/src/session.c
+++ b/src/session.c
@@ -1,8 +1,8 @@
 /* Copyright (C) 2022 Simo Sorce <simo@redhat.com>
    SPDX-License-Identifier: Apache-2.0 */

-#include "provider.h"
 #include <string.h>
+#include "provider.h"

 /* Slot stuff */
 struct p11prov_slot {
diff --git a/src/util.c b/src/util.c
index ed996cf..0aad107 100644
--- a/src/util.c
+++ b/src/util.c
@@ -1,12 +1,13 @@
 /* Copyright (C) 2022 Simo Sorce <simo@redhat.com>
    SPDX-License-Identifier: Apache-2.0 */

-#include "provider.h"
 #include <string.h>
 #include <time.h>
-#include "platform/endian.h"
+#include <sys/types.h>
 #include <openssl/bn.h>
 #include <openssl/x509.h>
+#include "provider.h"
+#include "platform/endian.h"

 #define FA_RETURN_VAL(x, _a, _b) \
     do { \
simo5 commented 1 year ago

The log seem to show that the provider fails during initialization, possibly related to failure to load the pkcs11 module.

Can you provide the contents of tests/tmp.softokn (specifically the openssl configuration file and the debug file).

On the patch above, provider.h is always supposed to be first as it include config.h and other standard headers as well as defined _XOPEN_SOURCE 500 which modifies how some system headers behave.

mouse07410 commented 1 year ago

Can you provide the contents of tests/tmp.softokn (specifically the openssl configuration file and the debug file).

$ cat tmp.softokn/openssl.cnf 
HOME = .

# Use this in order to automatically load providers.
openssl_conf = openssl_init

config_diagnostics = 1

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect
base = base_sect

[base_sect]
activate = 1

[default_sect]
activate = 1

[pkcs11_sect]
module = /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.so
pkcs11-module-init-args = configDir=/Users/ur20980/src/pkcs11-provider/tests/tmp.softokn/tokens
pkcs11-module-token-pin = file:/Users/ur20980/src/pkcs11-provider/tests/pinfile.txt
#pkcs11-module-allow-export
activate = 1

####################################################################
[ req ]
default_bits        = 2048
default_md      = sha256
default_keyfile     = privkey.pem
distinguished_name  = req_distinguished_name
attributes      = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert
string_mask = utf8only
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
countryName         = Country Name (2 letter code)
countryName_default     = US
countryName_min         = 2
countryName_max         = 2
stateOrProvinceName     = State or Province Name (full name)
stateOrProvinceName_default = New York
localityName            = Locality Name (eg, city)
localityName_default        = New York
0.organizationName      = Organization Name (eg, company)
0.organizationName_default  = PKCS11 Provider
organizationalUnitName      = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Testing Harness
commonName          = Common Name (eg, your name or your server\'s hostname)
commonName_max          = 64
emailAddress            = Email Address
emailAddress_max        = 64

[ req_attributes ]
challengePassword       = A challenge password
challengePassword_min       = 4
challengePassword_max       = 20
unstructuredName        = An optional company name

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]
# Extensions for a typical CA
# PKIX recommendation.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true
# Key usage: this is typical for a CA certificate. However since it will
# prevent it being used as an test self-signed certificate it is best
# left out by default.
# keyUsage = cRLSign, keyCertSign
# Include email address in subject alt name: another PKIX recommendation
subjectAltName=email:copy
# Copy issuer details
issuerAltName=issuer:copy
$  
$ cat tmp.softokn/gdb-commands.txt 
# r pkey -in $BASEURI -pubin -pubout -out ${TSTOUT}.pub
# r  x509 -in ${CRTURI} -subject -out ${TMPPDIR}/crt-subj.txt
# r  pkeyutl -sign -inkey "${ECBASEURI}" -digest sha256 -in ${RAND64FILE} -rawin -out ${TMPPDIR}/sha256-ecdgstsig.bin
# r  pkeyutl -encrypt -inkey "${BASEURI}" -pubin -pkeyopt pad-mode:oaep -pkeyopt digest:sha256 -pkeyopt mgf1-digest:sha256 -in ${SECRETFILE} -out ${SECRETFILE}.enc
# r  pkeyutl -derive -kdf HKDF -kdflen 48 -pkeyopt md:SHA256 -pkeyopt mode:EXTRACT_AND_EXPAND -pkeyopt hexkey:${HKDF_HEX_SECRET} -pkeyopt hexsalt:${HKDF_HEX_SALT} -pkeyopt hexinfo:${HKDF_HEX_INFO} -out ${TMPPDIR}/hkdf1-out-pkcs11.bin -propquery provider=pkcs11
# r  pkeyutl -sign -inkey "${BASEURI}" -digest sha256 -pkeyopt pad-mode:pss -pkeyopt mgf1-digest:sha256 -pkeyopt saltlen:digest -in ${RAND64FILE} -rawin -out ${TMPPDIR}/sha256-dgstsig.bin
$

. . . and the debug file

I have no clue what you mean by "the debug file".

On the patch above, provider.h is always supposed to be first . . .

Changed, making sure provider.h is the first include:

/bin/sh ../libtool  --tag=CC   --mode=compile clang -DHAVE_CONFIG_H -I.    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -Wall -Werror -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT pkcs11_la-util.lo -MD -MP -MF .deps/pkcs11_la-util.Tpo -c -o pkcs11_la-util.lo `test -f 'util.c' || echo './'`util.c
libtool: compile:  clang -DHAVE_CONFIG_H -I. -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -Wall -Werror -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT pkcs11_la-util.lo -MD -MP -MF .deps/pkcs11_la-util.Tpo -c util.c  -fno-common -DPIC -o .libs/pkcs11_la-util.o
util.c:569:15: error: implicitly declaring library function 'snprintf' with type 'int (char *, unsigned long, const char *, ...)' [-Werror,-Wimplicit-function-declaration]
        err = snprintf(alloc_name, 32, "sup_attr_%016lx", attr);
              ^
util.c:569:15: note: include the header <stdio.h> or explicitly provide a declaration for 'snprintf'
util.c:646:11: error: implicitly declaring library function 'vsnprintf' with type 'int (char *, unsigned long, const char *, struct __va_list_tag *)' [-Werror,-Wimplicit-function-declaration]
    ret = vsnprintf(buf, size_hint, format, args);
          ^
util.c:646:11: note: include the header <stdio.h> or explicitly provide a declaration for 'vsnprintf'
2 errors generated.
make[2]: *** [pkcs11_la-util.lo] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

After putting #include <stdio.h> before provider.h in src/util.c that file compiled successfully. And the tests failed the same way:

$ time make && time make check
Making all in src
/Applications/Xcode.app/Contents/Developer/usr/bin/make  all-am
/bin/sh ../libtool  --tag=CC   --mode=compile clang -DHAVE_CONFIG_H -I.    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -Wall -Werror -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT pkcs11_la-util.lo -MD -MP -MF .deps/pkcs11_la-util.Tpo -c -o pkcs11_la-util.lo `test -f 'util.c' || echo './'`util.c
libtool: compile:  clang -DHAVE_CONFIG_H -I. -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -Wall -Werror -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT pkcs11_la-util.lo -MD -MP -MF .deps/pkcs11_la-util.Tpo -c util.c  -fno-common -DPIC -o .libs/pkcs11_la-util.o
libtool: compile:  clang -DHAVE_CONFIG_H -I. -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -Wall -Werror -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT pkcs11_la-util.lo -MD -MP -MF .deps/pkcs11_la-util.Tpo -c util.c -o pkcs11_la-util.o >/dev/null 2>&1
mv -f .deps/pkcs11_la-util.Tpo .deps/pkcs11_la-util.Plo
/bin/sh ../libtool  --tag=CC   --mode=link clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -Wall -Werror -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11  -module -shared -shrext .dylib -avoid-version -export-symbols "./provider.exports"  -o pkcs11.la -rpath /usr/local/lib/ossl-modules pkcs11_la-asymmetric_cipher.lo pkcs11_la-debug.lo pkcs11_la-encoder.lo pkcs11_la-digests.lo pkcs11_la-exchange.lo pkcs11_la-kdf.lo pkcs11_la-keymgmt.lo pkcs11_la-interface.lo pkcs11_la-objects.lo pkcs11_la-provider.lo pkcs11_la-session.lo pkcs11_la-signature.lo pkcs11_la-store.lo pkcs11_la-tls.lo pkcs11_la-util.lo -L/Users/ur20980/openssl-3/lib -lssl -lcrypto 
libtool: link: /opt/local/bin/gsed -e 's|^|_|' < ./provider.exports > .libs/pkcs11-symbols.expsym
libtool: link: clang -Wl,-undefined -Wl,dynamic_lookup -o .libs/pkcs11.dylib -bundle  .libs/pkcs11_la-asymmetric_cipher.o .libs/pkcs11_la-debug.o .libs/pkcs11_la-encoder.o .libs/pkcs11_la-digests.o .libs/pkcs11_la-exchange.o .libs/pkcs11_la-kdf.o .libs/pkcs11_la-keymgmt.o .libs/pkcs11_la-interface.o .libs/pkcs11_la-objects.o .libs/pkcs11_la-provider.o .libs/pkcs11_la-session.o .libs/pkcs11_la-signature.o .libs/pkcs11_la-store.o .libs/pkcs11_la-tls.o .libs/pkcs11_la-util.o   -L/Users/ur20980/openssl-3/lib -lssl -lcrypto  -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk   -Wl,-exported_symbols_list,.libs/pkcs11-symbols.expsym
ld: warning: -undefined dynamic_lookup may not work with chained fixups
libtool: link: ( cd ".libs" && rm -f "pkcs11.la" && ln -s "../pkcs11.la" "pkcs11.la" )
Making all in tests
make[1]: Nothing to be done for `all'.
make[1]: Nothing to be done for `all-am'.

real    0m1.130s
user    0m0.500s
sys 0m0.316s
Making check in src
Making check in tests
/Applications/Xcode.app/Contents/Developer/usr/bin/make  tsession tgenkey ttls tdigests treadkeys \
      helpers.sh setup-softhsm.sh setup-softokn.sh softhsm-proxy.sh test-wrapper tbasic tcerts teccsha2 thkdf toaepsha2 trsapss
clang -DHAVE_CONFIG_H -I. -I../src    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT tsession-tsession.o -MD -MP -MF .deps/tsession-tsession.Tpo -c -o tsession-tsession.o `test -f 'tsession.c' || echo './'`tsession.c
mv -f .deps/tsession-tsession.Tpo .deps/tsession-tsession.Po
/bin/sh ../libtool  --tag=CC   --mode=link clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11   -o tsession tsession-tsession.o -L/Users/ur20980/openssl-3/lib -lssl -lcrypto 
libtool: link: clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -o tsession tsession-tsession.o  -L/Users/ur20980/openssl-3/lib -lssl -lcrypto
clang -DHAVE_CONFIG_H -I. -I../src    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT tgenkey-tgenkey.o -MD -MP -MF .deps/tgenkey-tgenkey.Tpo -c -o tgenkey-tgenkey.o `test -f 'tgenkey.c' || echo './'`tgenkey.c
mv -f .deps/tgenkey-tgenkey.Tpo .deps/tgenkey-tgenkey.Po
/bin/sh ../libtool  --tag=CC   --mode=link clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11   -o tgenkey tgenkey-tgenkey.o -L/Users/ur20980/openssl-3/lib -lssl -lcrypto 
libtool: link: clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -o tgenkey tgenkey-tgenkey.o  -L/Users/ur20980/openssl-3/lib -lssl -lcrypto
clang -DHAVE_CONFIG_H -I. -I../src    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT ttls-ttls.o -MD -MP -MF .deps/ttls-ttls.Tpo -c -o ttls-ttls.o `test -f 'ttls.c' || echo './'`ttls.c
mv -f .deps/ttls-ttls.Tpo .deps/ttls-ttls.Po
/bin/sh ../libtool  --tag=CC   --mode=link clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11   -o ttls ttls-ttls.o -L/Users/ur20980/openssl-3/lib -lssl -lcrypto 
libtool: link: clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -o ttls ttls-ttls.o  -L/Users/ur20980/openssl-3/lib -lssl -lcrypto
clang -DHAVE_CONFIG_H -I. -I../src    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT tdigests-tdigests.o -MD -MP -MF .deps/tdigests-tdigests.Tpo -c -o tdigests-tdigests.o `test -f 'tdigests.c' || echo './'`tdigests.c
mv -f .deps/tdigests-tdigests.Tpo .deps/tdigests-tdigests.Po
/bin/sh ../libtool  --tag=CC   --mode=link clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11   -o tdigests tdigests-tdigests.o -L/Users/ur20980/openssl-3/lib -lssl -lcrypto 
libtool: link: clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -o tdigests tdigests-tdigests.o  -L/Users/ur20980/openssl-3/lib -lssl -lcrypto
clang -DHAVE_CONFIG_H -I. -I../src    -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -MT treadkeys-treadkeys.o -MD -MP -MF .deps/treadkeys-treadkeys.Tpo -c -o treadkeys-treadkeys.o `test -f 'treadkeys.c' || echo './'`treadkeys.c
mv -f .deps/treadkeys-treadkeys.Tpo .deps/treadkeys-treadkeys.Po
/bin/sh ../libtool  --tag=CC   --mode=link clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11   -o treadkeys treadkeys-treadkeys.o -L/Users/ur20980/openssl-3/lib -lssl -lcrypto 
libtool: link: clang -Wall -Wextra -Wwrite-strings -Wpointer-arith -Wno-missing-field-initializers -Wformat -Wshadow -Wno-unused-parameter -Werror=implicit-function-declaration -Werror=missing-prototypes -Werror=format-security -Werror=parentheses -Werror=implicit -Werror=strict-prototypes -fno-strict-aliasing -fno-delete-null-pointer-checks -fdiagnostics-show-option -I/Users/ur20980/openssl-3/include -O3 -std=gnu18 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -std=c11 -o treadkeys treadkeys-treadkeys.o  -L/Users/ur20980/openssl-3/lib -lssl -lcrypto
make[2]: Nothing to be done for `helpers.sh'.
make[2]: Nothing to be done for `setup-softhsm.sh'.
make[2]: Nothing to be done for `setup-softokn.sh'.
make[2]: Nothing to be done for `softhsm-proxy.sh'.
make[2]: Nothing to be done for `test-wrapper'.
make[2]: Nothing to be done for `tbasic'.
make[2]: Nothing to be done for `tcerts'.
make[2]: Nothing to be done for `teccsha2'.
make[2]: Nothing to be done for `thkdf'.
make[2]: Nothing to be done for `toaepsha2'.
make[2]: Nothing to be done for `trsapss'.
/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
LIBSPATH=/Users/ur20980/src/pkcs11-provider/src/.libs \
    TESTSSRCDIR=/Users/ur20980/src/pkcs11-provider/tests \
    TESTBLDDIR=/Users/ur20980/src/pkcs11-provider/tests \
    SOFTOKNPATH="/opt/local/lib/nss/" \
    /Users/ur20980/src/pkcs11-provider/tests/setup-softokn.sh > setup-softokn.log 2>&1
LIBSPATH=/Users/ur20980/src/pkcs11-provider/src/.libs \
    TESTSSRCDIR=/Users/ur20980/src/pkcs11-provider/tests \
    TESTBLDDIR=/Users/ur20980/src/pkcs11-provider/tests \
    P11KITCLIENTPATH="/opt/local/lib/pkcs11/p11-kit-client.so" \
    /Users/ur20980/src/pkcs11-provider/tests/setup-softhsm.sh > setup-softhsm.log 2>&1
FAIL: basic-softokn
SKIP: basic-softhsm-proxy
FAIL: certs-softokn
SKIP: certs-softhsm-proxy
FAIL: eccsha2-softokn
FAIL: oaepsha2-softokn
FAIL: hkdf-softokn
FAIL: rsapss-softokn
SKIP: digests-softokn
SKIP: digests-softhsm-proxy
FAIL: genkey-softokn
SKIP: genkey-softhsm
FAIL: session-softokn
SKIP: session-softhsm-proxy
FAIL: readkeys-softokn
SKIP: readkeys-softhsm-proxy
PASS: tls-softokn
SKIP: tls-softhsm-proxy
============================================================================
Testsuite summary for pkcs11-provider 0.1
============================================================================
# TOTAL: 18
# PASS:  1
# SKIP:  8
# XFAIL: 0
# FAIL:  9
# XPASS: 0
# ERROR: 0
============================================================================
See tests/test-suite.log
Please report to simo@redhat.com
============================================================================
make[3]: *** [test-suite.log] Error 1
make[2]: *** [check-TESTS] Error 2
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1

real    0m7.201s
user    0m4.523s
sys 0m1.529s

test-suite.log tls-softokn.log readkeys-softokn.log session-softhsm-proxy.log session-softokn.log genkey-softhsm.log genkey-softokn.log digests-softokn.log rsapss-softokn.log hkdf-softokn.log oaepsha2-softokn.log eccsha2-softokn.log certs-softokn.log basic-softokn.log setup-softhsm.log setup-softokn.log

gdb-commands.txt

$ cat tests/tmp.softokn/openssl.cnf
HOME = .

# Use this in order to automatically load providers.
openssl_conf = openssl_init

config_diagnostics = 1

[openssl_init]
providers = provider_sect

[provider_sect]
default = default_sect
pkcs11 = pkcs11_sect
base = base_sect

[base_sect]
activate = 1

[default_sect]
activate = 1

[pkcs11_sect]
module = /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.so
pkcs11-module-init-args = configDir=/Users/ur20980/src/pkcs11-provider/tests/tmp.softokn/tokens
pkcs11-module-token-pin = file:/Users/ur20980/src/pkcs11-provider/tests/pinfile.txt
#pkcs11-module-allow-export
activate = 1

####################################################################
[ req ]
default_bits        = 2048
default_md      = sha256
default_keyfile     = privkey.pem
distinguished_name  = req_distinguished_name
attributes      = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert
string_mask = utf8only
req_extensions = v3_req # The extensions to add to a certificate request

[ req_distinguished_name ]
countryName         = Country Name (2 letter code)
countryName_default     = US
countryName_min         = 2
countryName_max         = 2
stateOrProvinceName     = State or Province Name (full name)
stateOrProvinceName_default = New York
localityName            = Locality Name (eg, city)
localityName_default        = New York
0.organizationName      = Organization Name (eg, company)
0.organizationName_default  = PKCS11 Provider
organizationalUnitName      = Organizational Unit Name (eg, section)
organizationalUnitName_default  = Testing Harness
commonName          = Common Name (eg, your name or your server\'s hostname)
commonName_max          = 64
emailAddress            = Email Address
emailAddress_max        = 64

[ req_attributes ]
challengePassword       = A challenge password
challengePassword_min       = 4
challengePassword_max       = 20
unstructuredName        = An optional company name

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment

[ v3_ca ]
# Extensions for a typical CA
# PKIX recommendation.
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true
# Key usage: this is typical for a CA certificate. However since it will
# prevent it being used as an test self-signed certificate it is best
# left out by default.
# keyUsage = cRLSign, keyCertSign
# Include email address in subject alt name: another PKIX recommendation
subjectAltName=email:copy
# Copy issuer details
issuerAltName=issuer:copy
$ 
$ ll /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.so
ls: /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.so: No such file or directory
$ ll /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.dylib
-rwxr-xr-x  1 ur20980  staff  225848 Jan 23 21:37 /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.dylib*
$

The problem seems to be the insistence of this code that shared/dynamic libraries have suffix .so, while MacOS insists that it's .dylib. Making this configurable (like, e.g., OpenSSL does) should resolve at least this problem.

mouse07410 commented 1 year ago

This is the patch that allows successful compilation and passing of several tests:

diff --git a/src/provider.h b/src/provider.h
index 07b66c8..23a9e35 100644
--- a/src/provider.h
+++ b/src/provider.h
@@ -10,6 +10,7 @@
 #include <stdbool.h>

 #include "pkcs11.h"
+#include <sys/types.h>
 #include <openssl/core_dispatch.h>
 #include <openssl/core_object.h>
 #include <openssl/types.h>
diff --git a/src/util.c b/src/util.c
index ed996cf..1c3a9e2 100644
--- a/src/util.c
+++ b/src/util.c
@@ -1,10 +1,12 @@
 /* Copyright (C) 2022 Simo Sorce <simo@redhat.com>
    SPDX-License-Identifier: Apache-2.0 */

+#include <stdio.h>
 #include "provider.h"
+#include "platform/endian.h"
 #include <string.h>
 #include <time.h>
-#include "platform/endian.h"
+#include <sys/types.h>
 #include <openssl/bn.h>
 #include <openssl/x509.h>

diff --git a/tests/openssl.cnf.in b/tests/openssl.cnf.in
index 769402a..debf883 100644
--- a/tests/openssl.cnf.in
+++ b/tests/openssl.cnf.in
@@ -20,7 +20,7 @@ activate = 1
 activate = 1

 [pkcs11_sect]
-module = @libtoollibs@/pkcs11.so
+module = @libtoollibs@/pkcs11.dylib
 pkcs11-module-init-args = configDir=@testsblddir@/tmp.softokn/tokens
 pkcs11-module-token-pin = file:@testsblddir@/pinfile.txt
 #pkcs11-module-allow-export

p11prov-debug.log test-suite.log

mouse07410 commented 1 year ago

This fixes locating Macports-installed SoftHSMv2 on MacOS:

diff --git a/tests/setup-softhsm.sh b/tests/setup-softhsm.sh
index c523839..c258457 100755
--- a/tests/setup-softhsm.sh
+++ b/tests/setup-softhsm.sh
@@ -29,6 +29,7 @@ find_softhsm() {

 title SECTION "Searching for SoftHSM PKCS#11 library"
 find_softhsm \
+    /opt/local/lib/softhsm/libsofthsm2.so \
     /usr/local/lib/softhsm/libsofthsm2.so \
     /usr/lib64/pkcs11/libsofthsm2.so \
     /usr/lib/pkcs11/libsofthsm2.so \

However, from MacOS point of view there's something is wrong with how certtool is invoked:

$ cat tests/setup-softhsm.log 
########################################
## Searching for SoftHSM PKCS#11 library

Using softhsm path /opt/local/lib/softhsm/libsofthsm2.so
########################################
## Set up testing system

Slot 0 has a free/uninitialized token.
The token has been initialized and is reassigned to slot 476010260
Creating new Self Sign CA
Using slot 0 with a present token (0x1c5f5714)
Key pair generated:
Private Key Object; RSA 
  label:      caCert
  ID:         0000
  Usage:      decrypt, sign, unwrap
  Access:     sensitive, always sensitive, never extractable, local
Public Key Object; RSA 2048 bits
  label:      caCert
  ID:         0000
  Usage:      encrypt, verify, wrap
  Access:     local
usage:
   Create a keypair and cert: certtool c [options]
   Create a CSR:              certtool r outFileName [options]
   Verify a CSR:              certtool V infileName [options]
   Create a system Identity:  certtool C domainName [options]
   Import a certificate:      certtool i inFileName [options]
   Display a certificate:     certtool d inFileName [options]
   Import a CRL:              certtool I inFileName [options]
   Display a CRL:             certtool D inFileName [options]
   Display certs and CRLs in keychain: certtool y [options]
Options:
   k=keychainName
   c (create the keychain)
   p=passphrase (specify passphrase at keychain creation)
   o=outFileName (create cert command only)
   v (verbose)
   d (infile/outfile in DER format; default is PEM)
   r=privateKeyFileName (optional; for Import Certificate only)
   f=[18fo] (private key format = PKCS1/PKCS8/FIPS186; default is PKCS1
     (openssl) for RSA, openssl for DSA, PKCS8 for Diffie-Hellman,
     OpenSSL for ECDSA
   x=[asSm] (Extended Key Usage: a=Any; s=SSL Client; S=SSL Server; m=SMIME)
   a (create key with default ACL)
   u (create key with ACL limiting access to current UID)
   P (Don't create system identity if one already exists for specified domain)
   h(elp)
$

In fact, it invokes /usr/bin/certtool, which probably is not what this code wants. certtool on MacOS modifies Keychain. I think we need to disable/block attempts to execute certtool by tests on MacOS.

Update

Also, an interesting failure of the test harness:

$ pwd
/Users/ur20980/src/pkcs11-provider/tests
$ cat tls-softokn.log 
Executing ./ttls
Failed to create SSL Context
FAIL tls-softokn (exit status: 1)
$ cat tls-softokn.trs
:test-result: FAIL
:global-test-result: FAIL
:recheck: yes
:copy-in-global-log: yes
$ ./ttls
SSL Context works!
$

When PKCS11_PROVIDER_MODULE env var is set to /opt/local/lib/nss//libsoftokn3.so, ./ttls fails. If that env var is unset or set to /Users/ur20980/src/pkcs11-provider/src/.libs/pkcs11.dylib - it succeeds.

simo5 commented 1 year ago

Sounds like detection of shared object extensions and install/executable paths need to be improved on MacOS, I do not have a Mac, so I will seek some help to get it right. May take a while.

mouse07410 commented 1 year ago

Sounds like detection of shared object extensions and install/executable paths need to be improved on MacOS

Yes, that's one thing. I'm afraid, not the only one (e.g., order of include statements in .c files).

I do not have a Mac, so I will seek some help to get it right. May take a while.

I have a Mac - please feel free to use me as a resource to try and experiment with things.

simo5 commented 1 year ago

Well for starters we need configure code that finds out what is the shared object extension on the machine and/or use an existing variable to store it, then change all Makefiles and scripts to use the extension directly and make it conditional via hat variable.

neverpanic commented 1 year ago

I have a Mac, work with Simo and am the openssl maintainer for MacPorts. I'll take a look at this.

mouse07410 commented 1 year ago

I think we can merge #181 - now all tests pass, except for one:

$ make check
Making check in src
Making check in tests
/Applications/Xcode.app/Contents/Developer/usr/bin/make  tsession tgenkey ttls tdigests treadkeys \
      helpers.sh setup-softhsm.sh setup-softokn.sh softhsm-proxy.sh test-wrapper tbasic tcerts teccsha2 thkdf toaepsha2 trsapss
make[2]: `tsession' is up to date.
make[2]: `tgenkey' is up to date.
make[2]: `ttls' is up to date.
make[2]: `tdigests' is up to date.
make[2]: `treadkeys' is up to date.
make[2]: Nothing to be done for `helpers.sh'.
make[2]: Nothing to be done for `setup-softhsm.sh'.
make[2]: Nothing to be done for `setup-softokn.sh'.
make[2]: Nothing to be done for `softhsm-proxy.sh'.
make[2]: Nothing to be done for `test-wrapper'.
make[2]: Nothing to be done for `tbasic'.
make[2]: Nothing to be done for `tcerts'.
make[2]: Nothing to be done for `teccsha2'.
make[2]: Nothing to be done for `thkdf'.
make[2]: Nothing to be done for `toaepsha2'.
make[2]: Nothing to be done for `trsapss'.
/Applications/Xcode.app/Contents/Developer/usr/bin/make  check-TESTS
PASS: basic-softokn
PASS: basic-softhsm-proxy
PASS: certs-softokn
FAIL: certs-softhsm-proxy
PASS: eccsha2-softokn
PASS: oaepsha2-softokn
PASS: hkdf-softokn
PASS: rsapss-softokn
PASS: digests-softokn
PASS: digests-softhsm-proxy
PASS: genkey-softokn
PASS: genkey-softhsm
PASS: session-softokn
PASS: session-softhsm-proxy
PASS: readkeys-softokn
PASS: readkeys-softhsm-proxy
PASS: tls-softokn
PASS: tls-softhsm-proxy
============================================================================
Testsuite summary for pkcs11-provider 0.1
============================================================================
# TOTAL: 18
# PASS:  17
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See tests/test-suite.log
Please report to simo@redhat.com
============================================================================
make[3]: *** [test-suite.log] Error 1
make[2]: *** [check-TESTS] Error 2
make[1]: *** [check-am] Error 2
make: *** [check-recursive] Error 1
$ cat tests/test-suite.log
===============================================
   pkcs11-provider 0.1: tests/test-suite.log
===============================================

# TOTAL: 18
# PASS:  17
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: certs-softhsm-proxy
=========================

Executing /Users/ur20980/src/pkcs11-provider/tests/softhsm-proxy.sh ./tcerts

## Start the p11-kit server and check if it works
Using slot 0 with a present token (0x11)

## Check we can fetch certifiatce objects
openssl x509 -in ${CRTURI} -subject -out ${TMPPDIR}/crt-subj.txt
p11-kit: 'bound != NULL' not true at fixed0_C_CloseSession
p11-kit: 'bound != NULL' not true at fixed0_C_CloseSession
p11-kit: 'bound != NULL' not true at fixed0_C_Finalize
(p11-kit:86688) message: child 86694 died with sigsegv

Cert not found looking for subject=O=PKCS11 Provider, CN=My Test Cert
killing p11-kit server
FAIL certs-softhsm-proxy (exit status: 1)

and tests/tmp.softhsm/p11prov-debug.log: p11prov-debug.log

simo5 commented 1 year ago

We need to either disable the test on Mac or fix it, can't merge with failing tests.

mouse07410 commented 1 year ago

We need to either disable the test on Mac or fix it, can't merge with failing tests.

Given that pretty much everything seems to work now, and without this PR most of things/tests did not work (on MacOS) - I suggest disabling that test and merging. Because without this PR the provider just does not work on MacOS, CI or no CI.

mouse07410 commented 1 year ago

Also, it would be great to get some guidance on migration from libp11 to this provider. Especially since both the libp11 engine and this provider binaries are named pkcs11.dylib. Like, where to copy the provider .dylib to?

And it would be nice if README covered the necessary changes to openssl.cnf file.

simo5 commented 1 year ago

Also, it would be great to get some guidance on migration from libp11 to this provider. Especially since both the libp11 engine and this provider binaries are named pkcs11.dylib. Like, where to copy the provider .dylib to?

At leats on Fedora the openssl-pkcs11 engine puts the module under /usr/lib64/engines-3/pkcs11.so while the provider is installed under /usr/lib64/ossl-modules/pkcs11.so so there should be no conflict.

And it would be nice if README covered the necessary changes to openssl.cnf file.

I would accept a PR contributing this, you can see what is needed in the openssl.conf.in tests file.

neverpanic commented 1 year ago

Fixed in #181.