After fresh install - failed to verify certificate: x509: certificate signed by unknown authority

[theLockesmith]

I had litd running just fine until I disabled it for a few weeks while I made some tweaks to lnd. I'm now trying to turn the service back up and I'm being met with this:

Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.739 [INF] LITD: LiT is running in mainnet, the taproot assets subserver do not support the `mainnet` network yet, disabling taproot assets subserver
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.739 [INF] LITD: LiT version: 0.10.2-alpha commit=v0.10.2-alpha
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.740 [INF] LITD: Listening for http_tls on: 127.0.0.1:8443
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.740 [INF] LITD: Listening for http on: [::]:8040
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.751 [INF] SESS: Checking for schema update: latest_version=0, db_version=0
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.752 [INF] LITD: Dialing lnd gRPC server at 127.0.0.1:10009
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.753 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:10009 127.0.0.1:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority". Reconnecting...
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.753 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:10009 127.0.0.1:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority". Reconnecting...
Jul 21 11:15:24 host litd[49929]: ----------------------------------------------------------
Jul 21 11:15:24 host litd[49929]:  Lightning Terminal (LiT) by Lightning Labs
Jul 21 11:15:24 host litd[49929]:
Jul 21 11:15:24 host litd[49929]:  LND Operating mode      remote
Jul 21 11:15:24 host litd[49929]:  LND Node status         locked
Jul 21 11:15:24 host litd[49929]:  LND Alias               ???? (node is locked)
Jul 21 11:15:24 host litd[49929]:  LND Version             ???? (node is locked)
Jul 21 11:15:24 host litd[49929]:  LiT Version             0.10.2-alpha commit=v0.10.2-alpha
Jul 21 11:15:24 host litd[49929]:  Web interface           127.0.0.1:8443, 0.0.0.0:8040 (open https://127.0.0.1:8443 or http://localhost:8040 in your browser)
Jul 21 11:15:24 host litd[49929]: ----------------------------------------------------------
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.753 [INF] LITD: Connecting basic lnd client
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.954 [INF] LITD: Connecting full lnd client
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.954 [INF] LNDC: Creating lnd connection to 127.0.0.1:10009
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.955 [INF] LNDC: Connected to lnd
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.957 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:10009 127.0.0.1:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority". Reconnecting...
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.958 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:10009 127.0.0.1:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority". Reconnecting...
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.958 [ERR] LITD: Could not set up LND clients: %!w(*errors.errorString=&{could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"})
Jul 21 11:15:24 host litd[49929]: 2023-07-21 11:15:24.958 [ERR] LITD: Error starting Lightning Terminal: could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
Jul 21 11:15:24 host litd[49929]: could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"

Here is my config:

network=mainnet

lnd-mode=remote
remote.lnd.rpcserver=127.0.0.1:10009
remote.lnd.macaroonpath=/home/user/data/chain/bitcoin/mainnet/admin.macaroon
remote.lnd.tlscertpath=/home/user/tls.cert

faraday-mode=integrated
pool-mode=integrated
loop-mode=integrated

httpslisten=0.0.0.0:8443
uipassword=supersecretpassword-pleasedonthack

So far I've

• recreated lnd tls.cert and tls.key
• recreated all macaroons
• deleted my .lit directory
• updated litd to the latest version

The other services I have running against port 10009 are giving no other indications of a problem. Is this an issue with my config or is something else at play here?

[guggero]

Are you sure these paths are correct?

remote.lnd.macaroonpath=/home/user/data/chain/bitcoin/mainnet/admin.macaroon
remote.lnd.tlscertpath=/home/user/tls.cert

It looks like you removed your username for privacy, but just to confirm: Is the configured path /home/<your_user>/tls.cert or /home/<your_user>/.lnd/tls.cert? If it's the first one, what does echo $HOME say? Could be that litd puts its own TLS certificate at the same place and overwrites lnd's?

[theLockesmith]

It is /home/<my user>/tls.cert and I can confirm that when litd creates a new cert, it is being generated in the /home/<my user>/.litd directory. echo $HOME resolves to /home/<my user>.

I don't remember making any changes, but these things don't usually just change on their own, so I'm sure I did something. I'm now getting the following error instead of the signed by unknown authority:

[ERR] LITD: Could not set up LND clients: %!w(*errors.errorString=&{could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2023-07-24T03:53:28-04:00 is after 2023-07-22T13:31:31Z"})

After checking both certs the program looks to be referring to with openssl x509 -in ~/.lit/tls.cert -text -noout (and again with the path to the lnd cert), it appears the cert in question for this particular error is the litd cert, but I'm not sure why as the date falls within the validity window:

Validity
            Not Before: Jul 22 19:47:08 2023 GMT
            Not After : Sep 15 19:47:08 2024 GMT

[guggero]

Hmm, very weird. So if you run lncli --rpcserver 127.0.0.1:10009 --tlscertpath /home/<your_user>/tls.cert --macaroonpath /home/<your_user>/data/chain/bitcoin/mainnet/admin.macaroon getinfo do you also get an error or does that work fine? If you do get an error, then re-generating the TLS cert for lnd didn't work. Make sure to delete both the key and cert file, then restart lnd to re-create it.

[theLockesmith]

Yes, that command works just fine. Though, I did go through a patch while adjusting my lnd.conf where I was getting the x509 error running that command, but that resolved itself once I deleted my lnd tls.cert and tls.key and regenerated them.

For posterity sake, I deleted my lnd tls.cert and tls.key just now and restarted lnd, and I'm back to the x509 error. And my lncli is working again. Though, I don't know if it's helpful or related, when I do that, I have to edit my lncli profiles.json file and update both the macaroon and tls.cert in that file for it to work properly again.

[guggero]

Ah, you're using profiles with lncli, good to know. So it makes sense that you first need to update the profiles file after re-generating the certificate. This is very weird indeed, I'm starting to run out of ideas to debug this. So the file is correct but litd still complains. Can you check if the file is a symlink or something unexpected? Are you running either of the two binaries in a container of any kind? Are both on the same machine? I assume you restarted litd several times in the meantime, right?

[theLockesmith]

No symlinks. No containers (for either of these 2 applications) Both on same machine, and I've tried 127.0.0.1, localhost, machine IP, and local fqdn. I have a litd service, which I've restarted and stopped entirely to try starting litd directly via command line. I've also tried starting litd using cli arguments with no config file.

The fact that the application regenerates the cert and key make me wonder if there's some kind of caching going on perhaps?

[guggero]

There shouldn't be any caching of certs, no... It's probably something small, but finding it sounds tough. I assume you made sure there's nothing else listening on port 10009.

[theLockesmith]

Hmmm I guess I'm not 100% sure of the inner workings of tor, but I don't believe it should be listening to traffic outside of the tor network. That being said, I do have a tor hidden service listening on port 10009 that is forwarding tor traffic to local 10009.

HiddenServicePort 10009 127.0.0.1:10009

Edit: I don't know why I didn't just check before posting the reply... I disabled that hidden service and am still getting the x509.

[guggero]

The hidden service also shouldn't matter. That's listening on Tor port 10009 (which is its separate interface) and just forwards traffic to the lnd port. I'm sure this will probably also lead nowhere. But can you run openssl s_client -connect 127.0.0.1:10009 and compare the certificate returned there with the file at /home/<your_user>/tls.cert?

[theLockesmith]

wow! they are not the same cert. That would make this an lnd problem I believe. Any ideas why it might be serving up an old cert?

[guggero]

Are you running Let's Encrypt or something similar? You said you re-generated the lnd certificate by removing both key+cert and then restarting lnd? So even after that the file and the cert are diverging? Can you post the relevant parts of your lnd config please? Interesting things are: lnd or data directory, anything related to certificates or paths, listening ports and so on.

[theLockesmith]

I'm not running letsencrypt on this machine. I did try it at one point, but quickly realized it was unnecessary, turned it off, and have since relied on lnd to generate the necessary certificates.

At this point I'm not going to take the time to scrub file paths. Here's my data/tls paths and listening ports:

datadir=/data/lightning/data
logdir=/data/lightning/logs
tlscertpath=/data/lightning/tls.cert
tlskeypath=/data/lightning/tls.key
tlsautorefresh=true
tlscertduration=10080h
tlsdisableautofill=true
tlsencryptkey=true

listen=0.0.0.0
rpclisten=0.0.0.0:10009
restlisten=0.0.0.0:18080

prometheus.enable=true
prometheus.listen=0.0.0.0:8989
watchtower.listen=0.0.0.0:9911

I don't see anything else related to tls or ports that isn't commented out. Obviously, my paths aren't actually /home/<user>, but /data/lightning and that is consistent across the config and the /home/<user>/.lnd doesn't exist.

edit: scrubbed some IPs

[guggero]

Ah, I see you're encrypting the TLS key. So I think that has some implications (as it can only use the encrypted key after unlocking the node). Can you try this please: Make sure the lnd node is unlocked. Then run openssl s_client -connect 127.0.0.1:10009 and copy the certificate (starting at -----BEGIN CERTIFICATE----- until -----END CERTIFICATE----- including those lines) into a new file (e.g. /data/lightning/tls-unlocked.cert) and then specify that file in litd?

[theLockesmith]

That was what I needed! Thank you so much for working through this with me!

[guggero]

Okay, great! Though I think we need to figure out how exactly that happened and whether it's reproducible with the encrypted key. Would you mind sending me the two TLS keys (the original tls.cert and the tls-unlocked.cert) in a Slack DM or by email?

[theLockesmith]

Absolutely! I'll send via the email you have listed on your profile if that works.

[guggero]

Thanks for the keys. I can confirm this has something to do with the encryption. Because the keys used to decrypt the certificate is only obtained after unlocking the node, lnd creates an ephemeral key (that is only valid for 2 days). After decrypting the key, it should switch over to the "long-term" TLS key that matches the certificate in tls.cert. But it looks like that did not happen (you can confirm you definitely unlocked the node before pulling the cert with openssl s_client right?).

Could you perhaps try if this was fixed by https://github.com/lightningnetwork/lnd/pull/7739 (would require to run lnd on the master branch).

In any case, I'm going to close the issue, as it's definitely not LiT related. If you can't test if this is fixed in master, could you please open an issue in lnd, giving the info in this comment (or just linking to it)?

Thanks for helping to debug this!

[theLockesmith]

I ran lncli unlock and was greeted with a password prompt followed by this: [lncli] rpc error: code = Unknown desc = wallet already unlocked, WalletUnlocker service is no longer available which, to the best of my knowledge, is just telling me that it is already unlocked and not indicative of anything else, and can confirm I did that before running the openssl s_client command.

I'll try pulling the master branch and see if that fixes it. I remember finding that issue while I was trying to fix this, but never went through the process. I guess I should have at least tried it!

[theLockesmith]

I get this now when using the unlocked cert:

2023-07-24 10:50:43.075 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.0.5.11:10009 10.0.5.11:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"4dxvnce4dxvncefyv4sf6hnkbqmz5l3zb4q3fa2ykiduusq3qudbirjukhloiyd.onion\")". Reconnecting...
2023-07-24 10:50:43.076 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.0.5.11:10009 10.0.5.11:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"4dxvnce4dxvncefyv4sf6hnkbqmz5l3zb4q3fa2ykiduusq3qudbirjukhloiyd.onion\")". Reconnecting...
2023-07-24 10:50:43.076 [ERR] LITD: Could not set up LND clients: %!w(*errors.errorString=&{could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"4dxvnce4dxvncefyv4sf6hnkbqmz5l3zb4q3fa2ykiduusq3qudbirjukhloiyd.onion\")"})
2023-07-24 10:50:43.076 [ERR] LITD: Error starting Lightning Terminal: could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"4dxvnce4dxvncefyv4sf6hnkbqmz5l3zb4q3fa2ykiduusq3qudbirjukhloiyd.onion\")"
could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"4dxvnce4dxvncefyv4sf6hnkbqmz5l3zb4q3fa2ykiduusq3qudbirjukhloiyd.onion\")"

That onion address is my advertise (externalhosts) address. At least I assume that's why it's the primary address listed on the cert.

And I still get the x509 when I try to use the locked cert:

2023-07-24 11:06:58.115 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.0.5.11:10009 10.0.5.11:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority". Reconnecting...
2023-07-24 11:06:58.117 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {10.0.5.11:10009 10.0.5.11:10009 <nil> 0 <nil>}. Err: connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority". Reconnecting...
2023-07-24 11:06:58.117 [ERR] LITD: Could not set up LND clients: %!w(*errors.errorString=&{could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"})
2023-07-24 11:06:58.117 [ERR] LITD: Error starting Lightning Terminal: could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"
could not create LND Services client: error subscribing to lnd wallet state: lnd version incompatible, need at least v0.13.0-beta, got error on state subscription: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority"