Support for MacOS

[braydenmerrifield]

Hi there,

I've recently been attempting to use the [kmsp11](https://github.com/GoogleCloudPlatform/kms-integrations/tree/master/kmsp11) module on MacOS to self sign a test certificate using my HSM key with no luck. I've been following the guide here, successfully setting the engine, and setting the module as directed.

However, the result I get is:

openssl engine pkcs11 -t
(pkcs11) pkcs11 engine
     [ available ]

openssl req -new -x509 -days 3650 -subj '/CN=test/' -sha512 -engine pkcs11 -keyform engine -key pkcs11:object=my-hsm-key  > my-request.crt
Engine "pkcs11" set.
dlopen(/opt/homebrew/Cellar/openssl@3/3.1.4/lib/ossl-modules/libkmsp11.so, 0x0005): tried: '/opt/homebrew/Cellar/openssl@3/3.1.4/lib/ossl-modules/libkmsp11.so' (not a mach-o file)

I assume this is because of the shared object file type.

Is there any chance of getting some guidance for MacOS (and compatible module?) as part of the kmsp11 user guide?

[bdhess]

Hey @braydenmerrifield - while we don't publish an official release for macOS, this project can be built/tested on Mac assuming you have Xcode tools installed, and our build tool (Bazel). The easiest way to get the right Bazel is with Bazelisk.

bazelisk test ... in the root of this repo will build the library and all run all its tests. When that completes successfully (about 6 minutes later on M1), you'll find the library at bazel-bin/kmsp11/main/libkmsp11.so.

[rani-sharim]

@braydenmerrifield How did you even get this to build on macos?

I'm getting either

./common/openssl.h:45:10: fatal error: 'openssl/libcrypto-compat.h' file not found

With a boringssl build, and

error: conflicting types for 'EVP_PKEY_get0_RSA'

with openssl build

[rkeene]

It's a bit of a stretch to say that this works with MacOS, given that the build system is so fragile. It may be possible, under some undefined circumstances, to get a binary that works on some MacOS system, but in my experience it just produces broken builds which may not be linked to other processes due to unresolved symbols.

[rkeene]

To be more specific here, it is possible to get the software contained in this repository to produce a PKCS#11 module which is usable on MacOS, but only if you are persistent and don't mind doing part of the work manually.

I've successfully produced a working PKCS#11 module from this code (as of 1f2649ed4536919be91730dccef84803f7ace6c3), but to do so I had to manually perform the linking since some dependencies did not get specified in the linker line (resulting in undefined symbols).

The result was a 56MiB Mach-O 64-bit arm64 dynamically linked shared library:

$ ls -lh libkmsp11.dylib 
-rwxr-xr-x 1 rkeene staff 51M May 16 15:42 libkmsp11.dylib
$ file libkmsp11.dylib 
libkmsp11.dylib: Mach-O 64-bit arm64 dynamically linked shared library, flags:<NOUNDEFS|DYLDLINK|TWOLEVEL|BINDS_TO_WEAK|NO_REEXPORTED_DYLIBS|HAS_TLV_DESCRIPTORS>

$ cat /tmp/kms-test/conf 
{
    "tokens": [{
        "key_ring": "<elided>",
        "label": "ssh"
    }]
}
$ KMS_PKCS11_CONFIG=/tmp/kms-test/conf ssh-agent -d -P "$(pwd -P)/*" & 
$ export SSH_AUTH_SOCK=<elided>
$ ssh-add -s "$(pwd -P)/libkmsp11.dylib"
$ ssh-add -L
ssh-rsa AAA...
$ ssh testuser@dev-test-ssh-sftp
  ########################[ Welcome ]########################
  #  You have logged in to the guest OS.                    #
  #  To access your containers use 'docker attach' command  #
  ###########################################################
...                                                          

Without manual modifications, the resulting library cannot be loaded:

$ bazel build --config macos  //kmsp11/main:libkmsp11.so
...
$ ssh-agent -d -P "$(readlink -f "$(pwd -P)/bazel-bin/kmsp11/main")/*" &
$ export SSH_AUTH_SOCK=<elided>
$ ssh-add -s "$(readlink -f "$(pwd -P)/bazel-bin/kmsp11/main/libkmsp11.so")"
...
debug1: process_add
dlopen <path>/libkmsp11.so failed: dlopen(<path>/libkmsp11.so, 0x0002): symbol not found in flat namespace '__Z11gpr_memrchrPKvim'
Could not add card "<path>/libkmsp11.so": agent refused operation