Terminal server shuts down from hash error

[gkrizek]

I have a node that is running Litd with everything integrated (lnd, loop, pool, faraday). One day I noticed that litd was still running but LNC sessions weren't working. After investigation, I found that litd and LND were still running but the Terminal Server (for LNC), Loop, Pool, and Faraday were all shut down. After looking at the logs I found the reason for the shutdown:

2023-08-22 00:04:06.413 [ERR] CHDB: unable to fetch amp htlcs for inv [0 0 0 86] and setIDs [0x40048c9f00]: %!w(*errors.errorString=&{unable to locate invoice})
2023-08-22 00:04:06.413 [ERR] ACCT: Error in invoice subscription: invalid hash length of 0, want 32
2023-08-22 00:04:06.413 [ERR] LITD: Received critical error from subsystem, shutting down: invalid hash length of 0, want 32

Expected behavior

No process in Litd should shut down because of this error. I believe that this error should be caught and returned in the APIs, but it's not a fatal error that would shut down the server.

Actual behavior

It shuts down some subservers

To reproduce

I don't know exactly how to replicate this. This is a user's node and I don't have all the information into what they do. They did say they never use the APIs directly, rather use things like Zeus, Thunderhub, and lncli. I'm not sure the exactly API call or function to cause this.

System information

litd 0.10.4

[gkrizek]

Had another unrelated customer hit this as well

[guggero]

I looked into this a bit and couldn't find an obvious reason for the hash to be empty. Could you give us a bit more information please? Would be great to know what exact user interface they are using that causes the shutdown. And the output of lncli listinvoices would be very helpful as well. Is this happening on every startup now or just after a certain action?

[gkrizek]

Yeah, sorry it's hard to provide more info because these are customers that report litd isn't working any more and I can only see the this info in the logs. Luckily one of the users affected here is a Lightning Labs employee so he said he can give an lncli listinvoices output.

It doesn't look like it happens right at start up. For example:

2023-09-01 10:03:13.601 [ERR] LITD: Received critical error from subsystem, shutting down: invalid hash length of 0, want 32
2023-08-31 23:06:06.897 [INF] LITD: Internal sub server startup complete

From talking with users, no one is doing something 'unique' it sounds like. Just using the standard APIs. I wonder if this is something from like Loop or Pool? I don't know I'm just guessing at this point.

[levmi]

This should be handled by a fix that we just merged into litd: https://github.com/lightninglabs/lightning-terminal/pull/642 So, when we cut the next release, which should be pretty soon here. Hopefully, this issue is fixed. Can re-open then if not for some unknown reason.

[kaloudis]

Still hitting this on master @ 8d7c53b2d1ccab6abc41a01b7c61e874cf194542

[kaloudis]

Can confirm it works with the addition of https://github.com/lightninglabs/lndclient/pull/156

[ViktorTigerstrom]

Hi @kaloudis! Just want to confirm, does the entire litd daemon shut down for you when you hit this, or just the account service (i.e. the other features and sub-servers of litd is still useable despite the account service being shutdown)?

We should probably have clarified this more clearly, but what we fixed with the latest release was that only the account service will shutdown when you hit this error, and not the entire litd daemon (i.e. achieving the expected behaviour in the description of this issue "No process in Litd should shut down because of this error."). Sorry for the confusion!

We're still working on adding AMP invoice support for the account service, so it's expected the behaviour with this release that this error will still cause the account service to shutdown.

I unfortunately do NOT recommend manually adding https://github.com/lightninglabs/lndclient/pull/156 to your local setup, as we intentionally didn't include that lndclient fix in litd yet. As if you do so, what will end up happening is that the account users that generated the AMP invoices, will still get the invoice returned when they generate the invoice, BUT any actual payments that are made using the account users AMP invoice will never be processed by the account system, meaning that the user will never get credited for any invoices that were settled using that AMP invoice. Unfortunately even after we add AMP invoice support, the invoices that were settled prior to us adding the support will never end up being processed. However once we add support for AMP invoices, any invoices that were settled using account users AMP invoices since the account service stopped working, will then be processed normally once litd is started.

If what you're experiencing is that only the account service is shutdown on startup due to hitting this error, but not litd itself, I'll re-close this issue again and open another issue for that just to make it more clearer that we still haven't added AMP invoice support :)!