Avoid force-closing channels when HtlcInterceptor request is not acknowledged

Background

Setting up a HtlcInterceptor to listen and interactively reject and accept incoming and outgoing payments can be a tool to improve the security of a node by auditing the forwards for arbitrary security policies, but ideally it would not interfere with the normal operation of the node by failing cleanly off-chain instead of causing force closes when events are not acknowledged.

Problem

LND will accept a startup without an HtlcInterceptor attached which somewhat limits its utility for security auditing but the main issue with general usage is that if the HtlcInterceptor stream or listener is interrupted the default behavior for any HTLC in-flight is to force close the channel and allow the pending HTLCs to time-out on chain.

This means that if any networking issue or bug causes an interruption in the gRPC control stream while a forward is pending an acknowledgement a costly force close will be the result.

Solution

This force-close behavior was also previously mirrored in the invoices RPC where held HTLCs that went unacknowledged would lead to force closes of relevant channels. But in LND 0.13 the height based expiry watcher was added to prevent this scenario by canceling back these held HTLCs at the last minute.

To provide the same benefit to forwarding HTLCs, a height watcher can perform a similar action on pending forwards and prevent force-closes deriving from errors in the HtlcInterceptor

Ideally this last-minute behavior is documented so that external block timeout assumption business logic doesn't cut things too close, but leaving timing down to the wire would be an anti-pattern in any case due to uncertainty of confirmation timing, and from a security assumptions perspective this behavior would also be consistent with the pre-existing invoices watcher.

Workaround

The workaround to held HTLCs in invoices was to regularly audit open invoices for held HTLCs and cancel them at the last minute before they caused a force close

I haven't tested this explicitly but it seems like in a somewhat similar way to watching for held invoices a caller could potentially reverse engineer the CircuitKey of an in-flight HTLC using the output of the channels list and setup a similar watcher to prevent force closes.

lightningnetwork / lnd