SIGSEGV running notary client binary

[nullterminated]

Downloaded notary client binary from

https://github.com/docker/notary/releases

for ubuntu 16.04. Verified checksum matches.

$ notary help fatal error: unexpected signal during runtime execution [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack: runtime.throw(0x9ae407, 0x2a) /usr/local/go/src/runtime/panic.go:566 +0x95 runtime.sigpanic() /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x2cc

goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x8045d0, 0xc420053e88, 0xc400000000) /usr/local/go/src/runtime/cgocall.go:131 +0x110 fp=0xc420053e58 sp=0xc420053e18 github.com/docker/notary/vendor/github.com/miekg/pkcs11._Cfunc_New(0x205afa0, 0x0) ??:0 +0x4a fp=0xc420053e88 sp=0xc420053e58 github.com/docker/notary/vendor/github.com/miekg/pkcs11.New(0x9abb57, 0x26, 0x0) /go/src/github.com/docker/notary/vendor/github.com/miekg/pkcs11/pkcs11.go:755 +0xa1 fp=0xc420053ec8 sp=0xc420053e88 github.com/docker/notary/trustmanager/yubikey.init.1() /go/src/github.com/docker/notary/trustmanager/yubikey/yubikeystore.go:98 +0xb6 fp=0xc420053f30 sp=0xc420053ec8 github.com/docker/notary/trustmanager/yubikey.init() /go/src/github.com/docker/notary/trustmanager/yubikey/yubikeystore.go:915 +0x91 fp=0xc420053f38 sp=0xc420053f30 github.com/docker/notary/client.init() /go/src/github.com/docker/notary/client/witness.go:70 +0x9b fp=0xc420053f40 sp=0xc420053f38 main.init() /go/src/github.com/docker/notary/cmd/notary/util.go:55 +0x4b fp=0xc420053f48 sp=0xc420053f40 runtime.main() /usr/local/go/src/runtime/proc.go:172 +0x1bf fp=0xc420053fa0 sp=0xc420053f48 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420053fa8 sp=0xc420053fa0

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

[HuKeping]

Is there a way that we can recreate the problem ourselves?

I tried on a Ubuntu 16.04 machine and seems fine.

But I guess it might because the vendor package that use the unsafe cgo, I'll have a check.

[HuKeping]

Could you please try this code on your machine:

package main

import (
    "fmt"
    "os"

    "github.com/miekg/pkcs11"
)

var possiblePkcs11Libs = []string{
    "/usr/lib/libykcs11.so",
    "/usr/lib64/libykcs11.so",
    "/usr/lib/x86_64-linux-gnu/libykcs11.so",
    "/usr/local/lib/libykcs11.so",
}

func main() {

    var pkcs11Lib string

    for _, loc := range possiblePkcs11Libs {
        _, err := os.Stat(loc)
        if err == nil {
            fmt.Println("loc is:", loc)
            p := pkcs11.New(loc)
            if p != nil {
                pkcs11Lib = loc
                return
            }
        } else {
            fmt.Println("err is:", err)
        }
    }

    fmt.Println("pkcs11lib is:", pkcs11Lib)

}

[nullterminated]

On a clean nuc with a recently installed 16.04, I can

curl -LO https://github.com/docker/notary/releases/download/v0.4.2/notary-Linux-amd64 && chmod +x notary-Linux-amd64 && ./notary-Linux-amd64 help

and get command info from the program. Looking at the trace, I suspect the problem lies in the fact that I have the latest yubico stable installed on my laptop.

https://launchpad.net/~yubico/+archive/ubuntu/stable

[riyazdf]

@nullterminated: thanks for your report! I've been digging into this on Ubuntu 16.04 - I'm able to reproduce the error with yubico-piv-tool versions 1.1.0+.

I didn't get the error on 1.0.3 - I think this is the version apt install yubico-piv-tool provides.

Still trying to determine what's causing the error itself, and I'll update this issue as I figure out more

Update on my findings: yubico-piv-tool 1.1.0+ introduced ykcs11 which is a partial implementation of PKCS#11.

My hunch is that there might be some missing functionality or difference that's causing the pkcs11 library we vendor to panic.

[riyazdf]

@a-dma: sorry to bother you, but I'm wondering if you're aware of any incompatibilities between ykcs11 and the pkcs11 library we're using, and any potential workarounds we might want to look into? We've noticed this panic since ykcs11 was introduced in version 1.1.0+ of yubico-piv-tool

[a-dma]

No, not that I'm aware of. The module itself (ykcs11) could do with improvements, but should not have "incompatibilities". Maybe try rebuilding it with debugging enabled and see if anything goes wrong in there.

[riyazdf]

@a-dma: will do. I've rebuilt ykcs11 with debugging but I'm not sure where I can see the debug output when testing integration with notary. Is there documentation that I could follow for how to use the debug mode (or a suggested place to look for logs)? Thanks!

[a-dma]

Nothing special, just build with ./configure --enable-ykcs11-debug and make. Debug output is printed on standard output, so that should end up wherever Notary prints it.

[riyazdf]

Ok thanks! I guess I'm not seeing any additional debug output after installing with ./configure --enable-ykcs11-debug make and sudo make install, so maybe this is happening pretty early when hooking into the library?

$ notary help -D
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x0]

runtime stack:
runtime.throw(0x9af2e7, 0x2a)
    /usr/local/go/src/runtime/panic.go:566 +0x95
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x2cc

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x805b20, 0xc420047e88, 0xc400000000)
    /usr/local/go/src/runtime/cgocall.go:131 +0x110 fp=0xc420047e58 sp=0xc420047e18
github.com/docker/notary/vendor/github.com/miekg/pkcs11._Cfunc_New(0x1fa4ff0, 0x0)
    ??:0 +0x4a fp=0xc420047e88 sp=0xc420047e58
github.com/docker/notary/vendor/github.com/miekg/pkcs11.New(0x9a5834, 0x1b, 0x0)
    /go/src/github.com/docker/notary/vendor/github.com/miekg/pkcs11/pkcs11.go:763 +0xc4 fp=0xc420047ec8 sp=0xc420047e88
github.com/docker/notary/trustmanager/yubikey.init.1()
    /go/src/github.com/docker/notary/trustmanager/yubikey/yubikeystore.go:98 +0xb6 fp=0xc420047f30 sp=0xc420047ec8
github.com/docker/notary/trustmanager/yubikey.init()
    /go/src/github.com/docker/notary/trustmanager/yubikey/yubikeystore.go:925 +0x91 fp=0xc420047f38 sp=0xc420047f30
github.com/docker/notary/client.init()
    /go/src/github.com/docker/notary/client/witness.go:70 +0x9b fp=0xc420047f40 sp=0xc420047f38
main.init()
    /go/src/github.com/docker/notary/cmd/notary/util.go:55 +0x4b fp=0xc420047f48 sp=0xc420047f40
runtime.main()
    /usr/local/go/src/runtime/proc.go:172 +0x1bf fp=0xc420047fa0 sp=0xc420047f48
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420047fa8 sp=0xc420047fa0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

[a-dma]

I don't know about Notary, but I made a quick test on my machine (running Ubuntu 16.10). I have run the snippet above and it works fine for me. Running that with a version of ykcs11 that has debugging enabled prints a couple of lines and then quits as expected. Keep in mind that there is an error in that snippet of code, the return in the innermost if should be a break.

My output is:

loc is: /usr/local/lib/libykcs11.so
debug: ykcs11.c:171 (C_GetFunctionList): In
debug: ykcs11.c:179 (C_GetFunctionList): Out
pkcs11lib is: /usr/local/lib/libykcs11.so

[HuKeping]

As per the stack

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x8045d0, 0xc420053e88, 0xc400000000)
/usr/local/go/src/runtime/cgocall.go:131 +0x110 fp=0xc420053e58 sp=0xc420053e18
github.com/docker/notary/vendor/github.com/miekg/pkcs11._Cfunc_New(0x205afa0, 0x0)
??:0 +0x4a fp=0xc420053e88 sp=0xc420053e58
github.com/docker/notary/vendor/github.com/miekg/pkcs11.New(0x9abb57, 0x26, 0x0)
/go/src/github.com/docker/notary/vendor/github.com/miekg/pkcs11/pkcs11.go:755 +0xa1 

Isn't it caused by the function New()

func New(module string) *Ctx {
        c := new(Ctx)
        mod := C.CString(module)
        defer C.free(unsafe.Pointer(mod))
        c.ctx = C.New(mod)
        if c.ctx == nil {
                return nil
        }    
        return c
}

At beginning, I was thinking if it was the module might be empty and so that the mod for C.New(mod) is nil that cause the panic. That's why I post the test code above, but it seems module was always be non-empty.

[cyli]

I can reproduce with the downloaded notary binary (which is a single static binary) as @riyazdf suggests with the ykcs11 deb and ubuntu 16.04. But if I compile the same version (v0.4.2) that is dynamically linked, I get no such segfault, although I do still get ykcs11 libary errors:

DEBU[0000] Configuration file not found, using defaults 
DEBU[0000] Using the following trust directory: /home/cyli/.notary 
ERRO[0000] could not reach https://notary-server:4443: Get https://notary-server:4443/v2/: dial tcp 127.0.0.1:4443: getsockopt: connection refused 
INFO[0000] continuing in offline mode                   
DEBU[0000] No yubikey found, using alternative key storage: found library /usr/lib/x86_64-linux-gnu/libykcs11.so, but initialize error pkcs11: 0x6: CKR_FUNCTION_FAILED 
DEBU[0000] No yubikey found, using alternative key storage: found library /usr/lib/x86_64-linux-gnu/libykcs11.so, but initialize error pkcs11: 0x6: CKR_FUNCTION_FAILED 
No root keys found. Generating a new root key...
DEBU[0000] generated ECDSA key with keyID: 4aaabd22a21d4e795b62918a4fb420748efe80be67b28caec22a9837b2a49e80 
DEBU[0000] generated new ecdsa key for role: root and keyID: 4aaabd22a21d4e795b62918a4fb420748efe80be67b28caec22a9837b2a49e80 
DEBU[0000] No yubikey found, using alternative key storage: found library /usr/lib/x86_64-linux-gnu/libykcs11.so, but initialize error pkcs11: 0x6: CKR_FUNCTION_FAILED 

That is the same presented error as https://github.com/docker/notary/issues/1006, but I'm not sure what service I might be running that'd be fighting for the card (no gpg, I've also apt-get removed pcscd)

[riyazdf]

ping @a-dma: @dhiltgen and I just ran into the panic on Ubuntu, we're wondering if you might have any more insight into the issue? Perhaps it could be something with https://github.com/miekg/pkcs11 ?

[noeljackson]

I'm having this exact same issue on Debian. Not to introduce more "me too" comments. But if there is any specific output you'd like from me, that would help, please let me know.

[florianeichin]

any updates on that issue?