session.login fails on MacOS Sonoma

[kristian1108]

I'm trying to figure out how to login to a PKCS11 session on my new operating system. This used to work. I've update to the latest version of this crate.

This is the line of code that's failing:

let result = session.login(UserType::User, Some(&secret_pin));

This is the error message that's printed:

Assertion failed: (ret == 0), function digest_final, file boringssl_crypto_digests.m, line 41.

    mov    x2, x8                   
    bl     0x104f28a88                ; symbol stub for: memcpy
    ldr    x0, [sp, #0x8]           
    ldr    x8, [sp, #0x58]          
    adrp   x1, 9843                 
    add    x1, x1, #0xf48             ; rustc_hex::CHARS + 15760
    bl     0x104559c54                ; <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual at result.rs:1960
    b      0x104560f50                ; <+536> at session_management.rs:61:6
    ldr    x8, [sp, #0x10]          
    ldr    x2, [sp, #0x28]          
    ldr    x1, [sp, #0x20]          
    ldr    x0, [sp, #0x18]          
    ldr    x9, [sp, #0x50]          
    ldr    x3, [x9, #0x8]           
    str    x3, [x9, #0x40]          
    blr    x8                       
    bl     0x1045557a0                ; <cryptoki::error::rv::Rv as core::convert::From<u64>>::from at rv.rs:19
    ldr    x8, [sp, #0x58]          
    bl     0x104556794                ; cryptoki::error::rv::Rv::into_result at rv.rs:131
    b      0x104560f50                ; <+536> at session_management.rs:61:6
    ldr    x8, [sp, #0x58]          
    adrp   x0, 9843                 
    add    x0, x0, #0xf30             ; rustc_hex::CHARS + 15736
    bl     0x104559f38                ; <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual at result.rs:1960
    b      0x104560f50                ; <+536> at session_management.rs:61:6
    ldp    x29, x30, [sp, #0x1b0]   
    ldp    x28, x27, [sp, #0x1a0]   
    add    sp, sp, #0x1c0           

Can add more detail as a find it, but if anyone has ideas that'd be super helpful. Thanks!

[kristian1108]

Update, here's a repo to reproduce the issue: https://github.com/kristian1108/boring-reproduce

[ionut-arm]

Hi,

I've tried running your code for reproducing, and it didn't crash for me - though this was on an M1 MacBook Air with MacOs Ventura. Let me upgrade to Sonoma, try again, and I'll get back to you on that.

[kristian1108]

Hi,

I've tried running your code for reproducing, and it didn't crash for me - though this was on an M1 MacBook Air with MacOs Ventura. Let me upgrade to Sonoma, try again, and I'll get back to you on that.

Thank you! Yes, I wouldn't expect it to crash on Ventura. Had no issue before the upgrade. Seems specific to Sonoma.

[ionut-arm]

Ok, can confirm I see the same... that's quite odd, not sure where it's coming from. Trying to access the SoftHSM using pkcs11-tools doesn't lead to the same issues. I'll leave this open and hopefully get back to it at some point

[kristian1108]

@ionut-arm did you downgrade back to Ventura after trying this out? I don't mean to be annoying, but it's basically blocking me from working on my project. I'm going to try to debug this myself, but am a little out in the deep end with my knowledge of these systems. If you have any suspicion of what's wrong, I'd love to leads/guidance on how to approach this. Thanks 🙏

[jippeholwerda]

I seem to remember having the same issue. It occurred after upgrading my Homebrew openssl version to 3.2.0. Somehow, that doesn't seem to be working well with the Homebrew version of SoftHSM.

I solved it by installing SoftHSM via the nix package manager. A team member solved it by compiling SoftHSM from source, pointing to the Homebrew openssl version: ./configure --with-openssl=/opt/homebrew/Cellar/openssl@3/3.2.0 Weirdly, that does work. No idea why.

Good luck!

[kristian1108]

@jippeholwerda I installed with Nix -- that fixed the session login. Thanks for the help there!!

But now my program is crashing with a segfault when it exits:

Caused by: process didn't exit successfully: /Users/kristian/code/program (signal: 11, SIGSEGV: invalid memory reference)

Did you see this at all when you were going through this?

Edit I've tracked this down to using Pkcs11 within a Lazy static. Updated the reproduction repo: https://github.com/kristian1108/boring-reproduce.

I may try to build from source next and see if that helps.

Edit 2 Building from source was the right answer. Instructions here: https://github.com/opendnssec/SoftHSMv2/tree/develop.

Thanks all!

[jippeholwerda]

@kristian1108 I didn't run into that problem, but if I run your boring-reproduce example I too get a segmentation fault.