Open dviator opened 6 years ago
Ugh. I would think this is likely an issue with libykcs11.dylib
as Notary itself doesn't call malloc
directly, so it may be worth opening an issue on https://github.com/Yubico/yubico-piv-tool as a starting point.
cc @a-dma - Apologies - would you happen to know what this was?
Not at a glance, too many abstraction layers.
If you say that Notary doesn't call malloc
then ykcs11
is the next likely suspect, but a tighter scoped/more detailed repro would be needed.
Also experiencing this as well:
Hey guys,
Been working on setting up a notary service at my company, and was planning on storing root keys on yubikey4C devices. Unfortunately performing signing with these yubikeys has been flaky. About 50% of the time it works, but the other 50% of the time it results in an error. I sometimes get different error outputs, though I'm guessing they are closely related, so I've included two of the error outputs below.
Context for my environment, I am using the notary installed with docker for mac: notary Version: 0.6.1 Git commit: d6e1431f
And the yubico-piv-tool installed by brew: yubico-piv-tool --version yubico-piv-tool 1.5.0
Error #1: notary -D init yubitesting200 DEBU[0000] Using the following trust directory: /Users/dveenstra/.notary DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session DEBU[0000] Found 1 objects matching list filters DEBU[0000] Making dir path: /Users/dveenstra/.notary/tuf/yubitesting200/changelist Root key found, using: ca0114a2fc49ee08c21753376a58a0a44f2d4851fb67ab6ee1dee33a462f8d27 DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session Please touch the attached Yubikey to perform signing. notary(24776,0x7000100dd000) malloc: error for object 0x4b02db0: incorrect checksum for freed object - object was probably modified after being freed. set a breakpoint in malloc_error_break to debug [1] 24776 abort notary -D init yubitesting200
Error #2: notary -D init yubitesting205 DEBU[0000] Using the following trust directory: /Users/dveenstra/.notary DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session DEBU[0000] Found 1 objects matching list filters DEBU[0000] Making dir path: /Users/dveenstra/.notary/tuf/yubitesting205/changelist Root key found, using: ca0114a2fc49ee08c21753376a58a0a44f2d4851fb67ab6ee1dee33a462f8d27 DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session DEBU[0000] Initialized PKCS11 library /usr/local/lib/libykcs11.dylib and started HSM session Please touch the attached Yubikey to perform signing. notary(24826,0x7fff95eaf380) malloc: error for object 0x4a0a980: incorrect checksum for freed object - object was probably modified after being freed. set a breakpoint in malloc_error_break to debug SIGABRT: abort PC=0x7fff5d4c0b6e m=0 sigcode=0
goroutine 0 [idle]: runtime: unknown pc 0x7fff5d4c0b6e stack: frame={sp:0x7ffeefbff2d8, fp:0x0} stack=[0x7ffeefb801e0,0x7ffeefbff660) 00007ffeefbff1d8: 00007fff5d531960 00007ffeefbff220 00007ffeefbff1e8: 00007fff5d67fd5e 0033000004946010 00007ffeefbff1f8: 0000000000000003 00000000049f4000 00007ffeefbff208: 00007ffeefbff898 00007ffeefbff320 00007ffeefbff218: 00007fff5d533a85 00007ffeefbff260 00007ffeefbff228: 00007fff5d52916d 00007fff95eaf380 00007ffeefbff238: 00007fff5d4bf773 00007fff95e9c2cc 00007ffeefbff248: 0000000004a0a980 0000000000000000 00007ffeefbff258: 00000000400000c0 00007ffeefbff340 00007ffeefbff268: 00007fff5d528c07 0000003000000020 00007ffeefbff278: 00000000049f3028 0000000000000000 00007ffeefbff288: 00000000049f3028 0000000000000000 00007ffeefbff298: 0000000000000000 0000000004a0a980 00007ffeefbff2a8: 00007fff5d534845 0000000000000000 00007ffeefbff2b8: 0000000004a0a8c0 00007ffeefbff438 00007ffeefbff2c8: 00007ffeefbff338 0000000000000001 00007ffeefbff2d8: <00007fff5d68b080 00007fff95eaf380 00007ffeefbff2e8: 00007ffeefbff328 0000000000000001 00007ffeefbff2f8: 00000000049f3000 0000000000000000 00007ffeefbff308: 00000000400000c0 00007ffeefbff340 00007ffeefbff318: 00007fff5d41c1ae 0000003000000010 00007ffeefbff328: 00007ffeffffffdf ffffffff04a0a980 00007ffeefbff338: 0000000004a0a980 00007ffeefbff460 00007ffeefbff348: 00007fff5d525ad4 0000000080100003 00007ffeefbff358: 0000000000000001 00007ffeefbff3d0 00007ffeefbff368: 00007fff3fc2dd2b 000000000494f600 00007ffeefbff378: 0000000004a0a8c0 000000000494f600 00007ffeefbff388: 0000000000000005 0000000004949080 00007ffeefbff398: 0000000004b00000 00007ffeefbff400 00007ffeefbff3a8: 00007fff5d531256 00007fff95d71cc0 00007ffeefbff3b8: 0000000004b06e90 00007ffeefbff400 00007ffeefbff3c8: 0000000000000001 00007ffeefbff3f0 runtime: unknown pc 0x7fff5d4c0b6e stack: frame={sp:0x7ffeefbff2d8, fp:0x0} stack=[0x7ffeefb801e0,0x7ffeefbff660) 00007ffeefbff1d8: 00007fff5d531960 00007ffeefbff220 00007ffeefbff1e8: 00007fff5d67fd5e 0033000004946010 00007ffeefbff1f8: 0000000000000003 00000000049f4000 00007ffeefbff208: 00007ffeefbff898 00007ffeefbff320 00007ffeefbff218: 00007fff5d533a85 00007ffeefbff260 00007ffeefbff228: 00007fff5d52916d 00007fff95eaf380 00007ffeefbff238: 00007fff5d4bf773 00007fff95e9c2cc 00007ffeefbff248: 0000000004a0a980 0000000000000000 00007ffeefbff258: 00000000400000c0 00007ffeefbff340 00007ffeefbff268: 00007fff5d528c07 0000003000000020 00007ffeefbff278: 00000000049f3028 0000000000000000 00007ffeefbff288: 00000000049f3028 0000000000000000 00007ffeefbff298: 0000000000000000 0000000004a0a980 00007ffeefbff2a8: 00007fff5d534845 0000000000000000 00007ffeefbff2b8: 0000000004a0a8c0 00007ffeefbff438 00007ffeefbff2c8: 00007ffeefbff338 0000000000000001 00007ffeefbff2d8: <00007fff5d68b080 00007fff95eaf380 00007ffeefbff2e8: 00007ffeefbff328 0000000000000001 00007ffeefbff2f8: 00000000049f3000 0000000000000000 00007ffeefbff308: 00000000400000c0 00007ffeefbff340 00007ffeefbff318: 00007fff5d41c1ae 0000003000000010 00007ffeefbff328: 00007ffeffffffdf ffffffff04a0a980 00007ffeefbff338: 0000000004a0a980 00007ffeefbff460 00007ffeefbff348: 00007fff5d525ad4 0000000080100003 00007ffeefbff358: 0000000000000001 00007ffeefbff3d0 00007ffeefbff368: 00007fff3fc2dd2b 000000000494f600 00007ffeefbff378: 0000000004a0a8c0 000000000494f600 00007ffeefbff388: 0000000000000005 0000000004949080 00007ffeefbff398: 0000000004b00000 00007ffeefbff400 00007ffeefbff3a8: 00007fff5d531256 00007fff95d71cc0 00007ffeefbff3b8: 0000000004b06e90 00007ffeefbff400 00007ffeefbff3c8: 0000000000000001 00007ffeefbff3f0
goroutine 1 [syscall]: runtime.cgocall(0x44391b0, 0xc42019a818, 0x0) /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc42019a7e8 sp=0xc42019a7b0 pc=0x4003dd4 github.com/theupdateframework/notary/vendor/github.com/miekg/pkcs11._Cfunc_Finalize(0x4a0a1b0, 0x0) _cgo_gotypes.go:812 +0x4a fp=0xc42019a818 sp=0xc42019a7e8 pc=0x43b5caa github.com/theupdateframework/notary/vendor/github.com/miekg/pkcs11.(Ctx).Finalize.func1(0x4a0a1b0, 0xc42019a870) /go/src/github.com/theupdateframework/notary/vendor/github.com/miekg/pkcs11/pkcs11.go:863 +0x56 fp=0xc42019a850 sp=0xc42019a818 pc=0x43c0b16 github.com/theupdateframework/notary/vendor/github.com/miekg/pkcs11.(Ctx).Finalize(0xc4200c61b0, 0xc42019a8d8, 0x43baa61) /go/src/github.com/theupdateframework/notary/vendor/github.com/miekg/pkcs11/pkcs11.go:863 +0x37 fp=0xc42019a890 sp=0xc42019a850 pc=0x43b9447 github.com/theupdateframework/notary/trustmanager/yubikey.finalizeAndDestroy(0x45bd620, 0xc4200c61b0) /go/src/github.com/theupdateframework/notary/trustmanager/yubikey/yubikeystore.go:817 +0x35 fp=0xc42019a8e8 sp=0xc42019a890 pc=0x43ce555 github.com/theupdateframework/notary/trustmanager/yubikey.cleanup(0x45bd620, 0xc4200c61b0, 0x51b660) /go/src/github.com/theupdateframework/notary/trustmanager/yubikey/yubikeystore.go:813 +0x66 fp=0xc42019a940 sp=0xc42019a8e8 pc=0x43ce476 github.com/theupdateframework/notary/trustmanager/yubikey.(YubiPrivateKey).Sign(0xc420368300, 0x45b4280, 0xc4200b4de0, 0xc42022c420, 0x20, 0x20, 0x45b50a0, 0xc4203a6fd8, 0xc42021d880, 0x40, ...) /go/src/github.com/theupdateframework/notary/trustmanager/yubikey/yubikeystore.go:208 +0x2c8 fp=0xc42019aa20 sp=0xc42019a940 pc=0x43c7d88 crypto/x509.CreateCertificate(0x45b4280, 0xc4200b4de0, 0xc4204a8680, 0xc4204a8680, 0x44eeb00, 0xc4204858e0, 0x4505860, 0xc420368300, 0xc4204a8680, 0x0, ...) /usr/local/go/src/crypto/x509/x509.go:2133 +0x8f7 fp=0xc42019b478 sp=0xc42019aa20 pc=0x429c627 github.com/theupdateframework/notary/cryptoservice.generateCertificate(0x45b6cc0, 0xc420368300, 0x7ffeefbff8a7, 0xe, 0xbec6ca28a596be01, 0x332a6b22, 0x4839560, 0xc379cae8a596be01, 0x460623ffb886b22, 0x4839560, ...) /go/src/github.com/theupdateframework/notary/cryptoservice/certificate.go:30 +0x245 fp=0xc42019b598 sp=0xc42019b478 pc=0x4399f95 github.com/theupdateframework/notary/cryptoservice.GenerateCertificate(0x45bb840, 0xc420368280, 0x7ffeefbff8a7, 0xe, 0xbec6ca28a596be01, 0x332a6b22, 0x4839560, 0xc379cae8a596be01, 0x460623ffb886b22, 0x4839560, ...) /go/src/github.com/theupdateframework/notary/cryptoservice/certificate.go:21 +0xd2 fp=0xc42019b630 sp=0xc42019b598 pc=0x4399c32 github.com/theupdateframework/notary/client.rootCertKey(0x7ffeefbff8a7, 0xe, 0x45bb840, 0xc420368280, 0xc420485200, 0xc42020b8f0, 0x1, 0x1) /go/src/github.com/theupdateframework/notary/client/client.go:169 +0xd9 fp=0xc42019b700 sp=0xc42019b630 pc=0x43d0819 github.com/theupdateframework/notary/client.(repository).createNewPublicKeyFromKeyIDs(0xc42012a4d0, 0xc42020b8d0, 0x1, 0x1, 0xc4200beea0, 0x1a0, 0xc42019b818, 0x4034a1b, 0x4013b7c) /go/src/github.com/theupdateframework/notary/client/client.go:276 +0x140 fp=0xc42019b7a8 sp=0xc42019b700 pc=0x43d1710 github.com/theupdateframework/notary/client.(repository).initialize(0xc42012a4d0, 0xc42020b8d0, 0x1, 0x1, 0xc42020b8e0, 0x0, 0x1, 0x0, 0x0, 0x0, ...) /go/src/github.com/theupdateframework/notary/client/client.go:222 +0x425 fp=0xc42019ba70 sp=0xc42019b7a8 pc=0x43d0e45 github.com/theupdateframework/notary/client.(repository).InitializeWithCertificate(0xc42012a4d0, 0xc42020b8d0, 0x1, 0x1, 0xc42020b8e0, 0x0, 0x1, 0x0, 0x0, 0x0, ...) /go/src/github.com/theupdateframework/notary/client/client.go:366 +0x400 fp=0xc42019bc10 sp=0xc42019ba70 pc=0x43d2380 main.(tufCommander).tufInit(0xc4200b7ad0, 0xc42034d200, 0xc420269200, 0x1, 0x2, 0x0, 0x0) /go/src/github.com/theupdateframework/notary/cmd/notary/tuf.go:505 +0x232 fp=0xc42019bcd8 sp=0xc42019bc10 pc=0x442f602 main.(tufCommander).(main.tufInit)-fm(0xc42034d200, 0xc420269200, 0x1, 0x2, 0x0, 0x0) /go/src/github.com/theupdateframework/notary/cmd/notary/tuf.go:139 +0x52 fp=0xc42019bd20 sp=0xc42019bcd8 pc=0x4435462 github.com/theupdateframework/notary/vendor/github.com/spf13/cobra.(Command).execute(0xc42034d200, 0xc4202691c0, 0x2, 0x2, 0xc42034d200, 0xc4202691c0) /go/src/github.com/theupdateframework/notary/vendor/github.com/spf13/cobra/command.go:698 +0x46d fp=0xc42019bdc8 sp=0xc42019bd20 pc=0x4171a0d github.com/theupdateframework/notary/vendor/github.com/spf13/cobra.(Command).ExecuteC(0xc4200d4fc0, 0x0, 0x0, 0x0) /go/src/github.com/theupdateframework/notary/vendor/github.com/spf13/cobra/command.go:783 +0x2e4 fp=0xc42019bef8 sp=0xc42019bdc8 pc=0x4172184 github.com/theupdateframework/notary/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200d4fc0, 0xc4200d4fc0, 0x0) /go/src/github.com/theupdateframework/notary/vendor/github.com/spf13/cobra/command.go:736 +0x2b fp=0xc42019bf28 sp=0xc42019bef8 pc=0x4171e7b main.main() /go/src/github.com/theupdateframework/notary/cmd/notary/main.go:196 +0x6a fp=0xc42019bf88 sp=0xc42019bf28 pc=0x442866a runtime.main() /usr/local/go/src/runtime/proc.go:198 +0x212 fp=0xc42019bfe0 sp=0xc42019bf88 pc=0x402d512 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42019bfe8 sp=0xc42019bfe0 pc=0x4057771
goroutine 19 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:139 +0xa7 os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
rax 0x0 rbx 0x7fff95eaf380 rcx 0x7ffeefbff2d8 rdx 0x0 rdi 0x307 rsi 0x6 rbp 0x7ffeefbff310 rsp 0x7ffeefbff2d8 r8 0x10 r9 0xfffffff0 r10 0x0 r11 0x206 r12 0x307 r13 0x49f3000 r14 0x6 r15 0x2d rip 0x7fff5d4c0b6e rflags 0x206 cs 0x7 fs 0x0 gs 0x0