clients fail SSL verification with sbd-serverd when given a valid certificate

i'm running into issues with SSL.

$ git rev-parse HEAD
0910518f0037d763d44a310ea0e52e1f46804f42
$ cargo run --bin hc_service_check -- signal -u wss://sbd-main-0.holo.host
    Finished dev [unoptimized + debuginfo] target(s) in 0.51s
     Running `target/debug/hc_service_check signal -u 'wss://sbd-main-0.holo.host'`
signal check of wss://sbd-main-0.holo.host
Custom { kind: InvalidData, error: InvalidCertificate(UnknownIssuer) }

for comparison i also brought up ngins SSL on port 444, and openssl seems happy with that.

$ openssl s_client -showcerts -connect sbd-main-0.holo.host:443 </dev/null > openssl-sbd.log
openssl s_client -showcerts -connect sbd-main-0.holo.host:444 </dev/null > openssl-nginx.log
diff openssl-nginx.log openssl-sbd.log
depth=0 CN = sbd-main-0.holo.host
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = sbd-main-0.holo.host
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = sbd-main-0.holo.host
verify return:1
DONE
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = sbd-main-0.holo.host
verify return:1
DONE

sbd is started with the following options, whereas cert.pem points to acme's fullchain.pem via systemd service configuration.

sbd-serverd --bind=65.108.241.120:443 --cert-pem-file=/run/credentials/sbd-server.service/cert.pem --priv-key-pem-file=/run/credentials/sbd-server.service/key.pem

33,66d32
<  1 s:C = US, O = Let's Encrypt, CN = R3
<    i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
<    a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
<    v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
< -----BEGIN CERTIFICATE-----
< MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
< TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
< cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
< WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
< RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
< AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
< R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
< sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
< NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
< Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
< /kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
< AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
< Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
< FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
< AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
< Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
< gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
< PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
< ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
< CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
< lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
< avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
< yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
< yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
< hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
< HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
< MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
< nLRbwHOoq7hHwg==
< -----END CERTIFICATE-----
77,78c43,44
< SSL handshake has read 2757 bytes and written 406 bytes
< Verification: OK
---
> SSL handshake has read 1447 bytes and written 406 bytes
> Verification error: unable to verify the first certificate
87c53
< Verify return code: 0 (ok)
---
> Verify return code: 21 (unable to verify the first certificate)

here's a snippet from the nginx config file:

ssl_certificate /var/lib/acme/sbd-main-0.holo.host/fullchain.pem;
ssl_certificate_key /var/lib/acme/sbd-main-0.holo.host/key.pem;
ssl_trusted_certificate /var/lib/acme/sbd-main-0.holo.host/chain.pem;

i'm just showing this config because sbd doesn't have an equivalent option to ssl_trusted_certificate. i don't know whether that's relevant here.

i did double-check that the certificates used by nginx and sbd are identical:

[root@sbd-0:~]# diff -s /var/lib/acme/sbd-main-0.holo.host/fullchain.pem <(nsenter -a -t $(pgrep sbd) cat /run/credentials/sbd-server.service/cert.pem)
Files /var/lib/acme/sbd-main-0.holo.host/fullchain.pem and /dev/fd/63 are identical

Originally posted by @steveej in https://github.com/holochain/holochain-infra/issues/96#issuecomment-2127689799

holochain / sbd

clients fail SSL verification with sbd-serverd when given a valid certificate #25