Panic crash on restoring in-memory circuit state from disk. Unable to start lnd

Horndev commented 6 years ago

Background

My LND node was found crashed. nothing unusual in the error logs.

When starting LND, I get a crash.

Your environment

version of lnd: 0.4.1 commit=097fd50747356a39593e5ac7625bf0b0fb1305a1 and upgraded to 0.4.2-beta commit=f0f5e11b826e020c11c37343bcbaf9725627378b with same error
which operating system (uname -a on *Nix): Ubuntu SMP Thu Feb 15 16:15:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
version of btcd, bitcoind, or other backend: Bitcoin Core Daemon version v0.16.0
any other relevant environment details

Steps to reproduce

Start lnd. Unlock wallet.

Log file:

2018-08-06 19:24:22.721 [WRN] LTND: open /home/horndev/.lnd/lnd.conf: no such file or directory
2018-08-06 19:24:22.721 [INF] LTND: Version 0.4.2-beta commit=f0f5e11b826e020c11c37343bcbaf9725627378b
2018-08-06 19:24:22.721 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2018-08-06 19:24:22.722 [INF] CHDB: Checking for schema update: latest_version=3, db_version=3
2018-08-06 19:24:22.745 [INF] RPCS: password RPC server listening on 10.0.0.4:8735
2018-08-06 19:24:22.745 [INF] RPCS: password gRPC proxy started at 10.0.0.4:8080
2018-08-06 19:24:22.745 [INF] LTND: Waiting for wallet encryption password. Use `lncli create` to create a wallet, `lncli unlock` to                 unlock an existing wallet, or `lncli changepassword` to change the password of an existing wallet and unlock it.
2018-08-06 19:24:42.332 [INF] LNWL: Opened wallet
2018-08-06 19:24:42.454 [INF] LTND: Primary chain is set to: bitcoin
2018-08-06 19:24:42.454 [INF] LTND: Initializing bitcoind backed fee estimator
2018-08-06 19:24:43.410 [INF] LNWL: Started listening for blocks via ZMQ on tcp://127.0.0.1:28332
2018-08-06 19:24:44.444 [INF] LNWL: The wallet has been unlocked without a time limit
2018-08-06 19:24:44.444 [INF] LTND: LightningWallet opened
2018-08-06 19:24:49.163 [INF] HSWC: Restoring in-memory circuit state from disk
2018-08-06 19:24:50.006 [INF] LTND: Shutdown complete
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb25049]

goroutine 1 [running]:
github.com/lightningnetwork/lnd/htlcswitch.(*circuitMap).restoreMemState.func1.2(0x7f869469a4e0, 0x10, 0x10, 0x7f869469a4f0, 0x10, 0                x10, 0x10, 0xc4201b6558)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/htlcswitch/circuit_map.go:284 +0x159
github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt.(*Bucket).ForEach(0xc4202ec000, 0xc42018ada8, 0x11, 0x11)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt/bucket.go:388 +0x109
github.com/lightningnetwork/lnd/htlcswitch.(*circuitMap).restoreMemState.func1(0xc4201b6540, 0x10e7500, 0xc4201b6540)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/htlcswitch/circuit_map.go:267 +0x137
github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt.(*DB).View(0xc4205ce780, 0xc42018aec0, 0x0, 0x0)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/vendor/github.com/coreos/bbolt/db.go:699 +0x90
github.com/lightningnetwork/lnd/htlcswitch.(*circuitMap).restoreMemState(0xc420177bc0, 0x0, 0x0)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/htlcswitch/circuit_map.go:238 +0xdf
github.com/lightningnetwork/lnd/htlcswitch.NewCircuitMap(0xc4201ac1a0, 0xc4201ac1a0, 0xe4f3a0, 0x0, 0x17674e0)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/htlcswitch/circuit_map.go:196 +0x7b
github.com/lightningnetwork/lnd/htlcswitch.New(0xc42012ce10, 0x1181a20, 0xc42000e1d8, 0xc4201ac180, 0xc420168d40, 0x11856a0, 0x17844                68, 0xc4201ac190, 0xc420604580, 0x118ee40, ...)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/htlcswitch/switch.go:251 +0xd9
main.newServer(0xc4201acad0, 0x1, 0x1, 0xc420168d40, 0xc4205001b0, 0xc42012ce10, 0x0, 0x0, 0x0)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/server.go:295 +0xbf9
main.lndMain(0x0, 0x0)
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/lnd.go:309 +0xad5
main.main()
        /home/horndev/gocode/src/github.com/lightningnetwork/lnd/lnd.go:451 +0x26

Expected behaviour

Start loading LND and connecting to peers

Actual behaviour

panic: runtime error: invalid memory address or nil pointer dereference

Horndev commented 6 years ago

For what it's worth, this is the log from the crash right before the issue started:

2018-08-06 17:51:57.692 [INF] HSWC: Sent 0 satoshis and received 301 satoshis in the last 10 seconds (0.200000 tx/sec)
2018-08-06 17:52:00.468 [INF] SRVR: New inbound connection from 188.254.162.84:54144
2018-08-06 17:52:01.328 [INF] CRTR: Updated vertex data for node=03c80be3937717b502e66dc6cf8ba2881c9e68a8398bfa08c2866babfc88a729f3
2018-08-06 17:52:12.097 [INF] DISC: Broadcasting batch of 9 new announcements
2018-08-06 17:52:15.361 [INF] PEER: NodeKey(03c80be3937717b502e66dc6cf8ba2881c9e68a8398bfa08c2866babfc88a729f3) loading ChannelPoint(ac6a74dfedd8a28c5fcefebec4ec5125c4e2fdec85cb0f25cce9f0fc7acc0435:0)
2018-08-06 17:52:15.361 [INF] NTFN: New block epoch subscription
2018-08-06 17:52:15.363 [INF] HSWC: ChannelLink(ac6a74dfedd8a28c5fcefebec4ec5125c4e2fdec85cb0f25cce9f0fc7acc0435:0) is starting
2018-08-06 17:52:15.363 [INF] CNCT: Attempting to update ContractSignals for ChannelPoint(ac6a74dfedd8a28c5fcefebec4ec5125c4e2fdec85cb0f25cce9f0fc7acc0435:0)
2018-08-06 17:52:15.363 [INF] HSWC: Added channel link with chan_id=3504cc7afcf0e9cc250fcb85ecfde2c42551ecc4befece5f8ca2d8eddf746aac, short_chan_id=((lnwire.ShortChannelID) 520547:1991:0
)
2018-08-06 17:52:15.363 [INF] HSWC: HTLC manager for ChannelPoint(ac6a74dfedd8a28c5fcefebec4ec5125c4e2fdec85cb0f25cce9f0fc7acc0435:0) started, bandwidth=0 mSAT
2018-08-06 17:52:15.363 [INF] HSWC: Trimming open circuits for chan_id=520547:1991:0, start_htlc_id=0
2018-08-06 17:52:15.363 [INF] HSWC: Attempting to re-resynchronize ChannelPoint(ac6a74dfedd8a28c5fcefebec4ec5125c4e2fdec85cb0f25cce9f0fc7acc0435:0)
2018-08-06 17:52:15.364 [INF] HSWC: Received re-establishment message from remote side for channel(ac6a74dfedd8a28c5fcefebec4ec5125c4e2fdec85cb0f25cce9f0fc7acc0435:0)
2018-08-06 17:52:15.570 [ERR] HSWC: unable to cancel HTLC: No HTLC with ID 1 in channel 520547:1991:0
2018-08-06 17:52:15.570 [ERR] HSWC: unable to cancel HTLC: No HTLC with ID 4 in channel 520547:1991:0
2018-08-06 17:52:15.570 [ERR] HSWC: unable to cancel HTLC: No HTLC with ID 5 in channel 520547:1991:0
2018-08-06 17:52:15.570 [ERR] HSWC: unable to cancel HTLC: No HTLC with ID 12 in channel 520547:1991:0
2018-08-06 17:52:15.570 [ERR] HSWC: unable to cancel HTLC: No HTLC with ID 22 in channel 520547:1991:0
2018-08-06 17:52:17.692 [INF] HSWC: Sent 0 satoshis and received 2462 satoshis in the last 10 seconds (83.600000 tx/sec)
2018-08-06 17:52:42.097 [INF] DISC: Broadcasting batch of 13 new announcements
2018-08-06 17:53:12.097 [INF] DISC: Broadcasting batch of 35 new announcements
2018-08-06 17:53:42.097 [INF] DISC: Broadcasting batch of 8 new announcements
2018-08-06 17:54:12.097 [INF] DISC: Broadcasting batch of 10 new announcements
2018-08-06 17:54:42.097 [INF] DISC: Broadcasting batch of 13 new announcements
2018/08/06 17:54:43 http: TLS handshake error from 58.218.56.87:4625: tls: first record does not look like a TLS handshake
2018-08-06 17:55:12.097 [INF] DISC: Broadcasting batch of 7 new announcements
2018-08-06 17:55:37.926 [ERR] HSWC: insufficient capacity in available outgoing links: need 850000 mSAT, max available is 0 mSAT
2018-08-06 17:55:37.926 [ERR] CRTR: Attempt to send payment b89784482a2849858d7b8c5027adce427daf3e364d68b2c78a09d699b8ac1aa7 failed: TemporaryChannelFailure: insufficient capacity in available outgoing links: need 850000 mSAT, max available is 0 mSAT

There is no information on why it crashed in the log. lnd was just dead and wouldn't start up again.

cfromknecht commented 6 years ago

@Horndev This patch should allow you to startup your node, so that we can get some more information. It does not modify the database, which we can once we're more certain of the db state.

Please make sure you set --nolisten on your node. We don't want to connect to peers rn, just want to start up and get some logs. Setting HSWC=debug log level will also help

diff --git a/htlcswitch/circuit_map.go b/htlcswitch/circuit_map.go
index 0e56d43e..ef686264 100644
--- a/htlcswitch/circuit_map.go
+++ b/htlcswitch/circuit_map.go
@@ -264,6 +264,7 @@ func (cm *circuitMap) restoreMemState() error {
            return ErrCorruptedCircuitMap
        }

+       var strayKeystones []Keystone
        if err := keystoneBkt.ForEach(func(k, v []byte) error {
            var (
                inKey  CircuitKey
@@ -280,15 +281,26 @@ func (cm *circuitMap) restoreMemState() error {

            // Retrieve the pending circuit, set its keystone, then
            // add it to the opened map.
-           circuit := pending[inKey]
-           circuit.Outgoing = outKey
-           opened[*outKey] = circuit
+           circuit, ok := pending[inKey]
+           if ok {
+               circuit.Outgoing = outKey
+               opened[*outKey] = circuit
+           } else {
+               strayKeystones = append(strayKeystones, Keystone{
+                   InKey:  inKey,
+                   OutKey: *outKey,
+               })
+           }

            return nil
        }); err != nil {
            return err
        }

+       for _, keystone := range strayKeystones {
+           log.Infof("Ignoring stray keystone: %v", keystone)
+       }
+
        return nil

    }); err != nil {

Horndev commented 6 years ago

Here is the output:

2018-08-06 22:21:29.260 [INF] HSWC: Restoring in-memory circuit state from disk
2018-08-06 22:21:29.311 [INF] LNWL: Started rescan from block 00000000000000000008aab7bf75d50969e64b6c753f90465b74ebec04fb4952 (height 535517) for 2538 addresses
2018-08-06 22:21:29.512 [INF] LNWL: Starting rescan from block 00000000000000000008aab7bf75d50969e64b6c753f90465b74ebec04fb4952
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=1)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=2)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=3)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=4)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=5)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=6)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=7)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=8)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=9)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=10)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=11)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=12)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=13)}
2018-08-06 22:21:30.076 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=14)}
2018-08-06 22:21:30.076 [INF] HSWC: Payment circuits loaded: num_pending=21328, num_open=167
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=521038:1472:0, start_htlc_id=0
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=531682:1285:0, start_htlc_id=0
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=520144:1560:0, start_htlc_id=0
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=529484:1779:0, start_htlc_id=0
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=519768:1525:0, start_htlc_id=172
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=520346:2274:0, start_htlc_id=601
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=514733:1220:1, start_htlc_id=1338
2018-08-06 22:21:30.270 [INF] HSWC: Trimming open circuits for chan_id=533217:1274:1, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=521416:473:1, start_htlc_id=894
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532879:833:1, start_htlc_id=7
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532879:839:1, start_htlc_id=85
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=527755:733:1, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=518528:1022:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=519048:1516:0, start_htlc_id=4
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532210:1582:1, start_htlc_id=59
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532286:1942:1, start_htlc_id=165
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=530245:303:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532997:1044:1, start_htlc_id=20
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=520745:490:1, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=529095:103:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532130:2121:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=519619:1375:1, start_htlc_id=815
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=521007:863:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=528983:977:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=519775:2039:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=515675:1474:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=529754:1325:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=516403:1187:0, start_htlc_id=494
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=519250:933:1, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=524757:1305:0, start_htlc_id=193
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=529310:524:1, start_htlc_id=24
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532880:1608:1, start_htlc_id=22
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=529537:1585:1, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=524639:109:0, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=520682:799:1, start_htlc_id=0
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=530773:2352:0, start_htlc_id=4
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=529255:2118:1, start_htlc_id=5
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=519605:1962:1, start_htlc_id=9
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=518451:1704:1, start_htlc_id=268
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=532879:835:1, start_htlc_id=40
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=529353:1209:0, start_htlc_id=26
2018-08-06 22:21:30.271 [INF] HSWC: Trimming open circuits for chan_id=527183:1708:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=533416:1549:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=519262:586:0, start_htlc_id=10
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=520045:945:0, start_htlc_id=147
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=530911:2042:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=519599:1789:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=532879:836:1, start_htlc_id=17
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=530264:990:0, start_htlc_id=1214
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=530628:680:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=528944:151:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=527697:239:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=528093:1521:1, start_htlc_id=107
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=519923:2165:1, start_htlc_id=4
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=532623:1141:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=532212:1117:1, start_htlc_id=1
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=528015:802:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=533680:681:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=517234:2226:1, start_htlc_id=501
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=522109:2045:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=521411:1295:1, start_htlc_id=165
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=532335:1691:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=533364:1363:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=529124:98:1, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=528060:2228:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=528261:1318:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=528437:1686:1, start_htlc_id=151
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=530289:2895:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=533364:1039:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=530216:1873:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=529379:1865:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=518219:565:0, start_htlc_id=0
2018-08-06 22:21:30.272 [INF] HSWC: Trimming open circuits for chan_id=532879:834:1, start_htlc_id=12
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=519070:1474:1, start_htlc_id=192
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=515078:377:1, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=533372:2216:1, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=520502:1967:0, start_htlc_id=571
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=518510:1692:0, start_htlc_id=172
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=520547:1991:0, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=532729:2354:0, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=533224:606:1, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=514614:1609:0, start_htlc_id=28
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=530867:913:0, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=529267:572:0, start_htlc_id=0
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=528437:1688:0, start_htlc_id=304
2018-08-06 22:21:30.273 [INF] HSWC: Trimming open circuits for chan_id=533310:655:1, start_htlc_id=0
2018-08-06 22:21:30.296 [INF] RPCS: RPC server listening on 10.0.0.4:8735
2018-08-06 22:21:30.296 [INF] RPCS: gRPC proxy started at 10.0.0.4:8080
2018-08-06 22:21:30.521 [INF] LTND: Waiting for chain backend to finish sync, start_height=535535
2018-08-06 22:21:36.401 [INF] LNWL: Rescan finished at 535535 (000000000000000000112ef42b7c3da06681a4f885ca448cc278ade931ebb05d)
2018-08-06 22:21:36.439 [INF] LNWL: Catching up block hashes to height 535535, this might take a while
2018-08-06 22:21:36.449 [INF] LNWL: Done catching up block hashes
2018-08-06 22:21:36.449 [INF] LNWL: Finished rescan for 2538 addresses (synced to block 000000000000000000112ef42b7c3da06681a4f885ca448cc278ade931ebb05d, height 535535)
2018-08-06 22:21:36.467 [INF] LNWL: Removed conflicting tx: (*wire.MsgTx)(0xc4216986c0)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=1 cap=1) {
  (*wire.TxIn)(0xc4216a8240)({
   PreviousOutPoint: (wire.OutPoint) 27e7d649f7214549a42baf12137123b7fa4b31968929f2f34952e44337e25e9b:0,
   SignatureScript: ([]uint8) {
   },

I trimmed a bunch of binary data (not sure if sensitive)

cfromknecht commented 6 years ago

This looks symptomatic of the issues fixed by https://github.com/lightningnetwork/lnd/pull/1166 and https://github.com/lightningnetwork/lnd/pull/1419, where the outgoing short channel ID is set to 0:0:0.

The following patch should actually remove the offending keystones, apply on most recent master.

diff --git a/htlcswitch/circuit_map.go b/htlcswitch/circuit_map.go
index 0e56d43e..ca588b30 100644
--- a/htlcswitch/circuit_map.go
+++ b/htlcswitch/circuit_map.go
@@ -235,7 +235,7 @@ func (cm *circuitMap) restoreMemState() error {
        pending = make(map[CircuitKey]*PaymentCircuit)
    )

-   if err := cm.cfg.DB.View(func(tx *bolt.Tx) error {
+   if err := cm.cfg.DB.Update(func(tx *bolt.Tx) error {
        // Restore any of the circuits persisted in the circuit bucket
        // back into memory.
        circuitBkt := tx.Bucket(circuitAddKey)
@@ -264,6 +264,7 @@ func (cm *circuitMap) restoreMemState() error {
            return ErrCorruptedCircuitMap
        }

+       var strayKeystones []Keystone
        if err := keystoneBkt.ForEach(func(k, v []byte) error {
            var (
                inKey  CircuitKey
@@ -280,15 +281,34 @@ func (cm *circuitMap) restoreMemState() error {

            // Retrieve the pending circuit, set its keystone, then
            // add it to the opened map.
-           circuit := pending[inKey]
-           circuit.Outgoing = outKey
-           opened[*outKey] = circuit
+           circuit, ok := pending[inKey]
+           if ok {
+               circuit.Outgoing = outKey
+               opened[*outKey] = circuit
+           } else {
+               strayKeystones = append(strayKeystones, Keystone{
+                   InKey:  inKey,
+                   OutKey: *outKey,
+               })
+           }

            return nil
        }); err != nil {
            return err
        }

+       for _, keystone := range strayKeystones {
+           if keystone.OutKey.ChanID != sourceHop {
+               continue
+           }
+
+           log.Infof("Removing stray keystone: %s", keystone)
+           err := keystoneBkt.Delete(keystone.OutKey.Bytes())
+           if err != nil {
+               return err
+           }
+       }
+
        return nil

    }); err != nil {

Horndev commented 6 years ago

Hi @cfromknecht , I tried this patch but it doesn't compile.

htlcswitch/circuit_map.go:302:23: invalid operation: keystone.OutKey != sourceHop (mismatched types channeldb.CircuitKey and lnwire.ShortChannelID)

cfromknecht commented 6 years ago

Sorry, fixed the diff. Try now?

Horndev commented 6 years ago

@cfromknecht OK, It compiled and I'm running it.

I cooperatively closed the one channel before launching.

Log output below from the startup:

2018-08-08 00:51:02.357 [INF] HSWC: Restoring in-memory circuit state from disk
2018-08-08 00:51:02.408 [INF] LNWL: Started rescan from block 00000000000000000005e9ad3ad1813710a3a71e8728585e112b7d4b8fc28a6d (height 535702) for 2550 addresses
2018-08-08 00:51:02.543 [INF] LNWL: Starting rescan from block 00000000000000000005e9ad3ad1813710a3a71e8728585e112b7d4b8fc28a6d
2018-08-08 00:51:02.545 [INF] LNWL: Rescan finished at 535702 (00000000000000000005e9ad3ad1813710a3a71e8728585e112b7d4b8fc28a6d)
2018-08-08 00:51:02.546 [INF] LNWL: Catching up block hashes to height 535702, this might take a while
2018-08-08 00:51:02.555 [INF] LNWL: Done catching up block hashes
2018-08-08 00:51:02.555 [INF] LNWL: Finished rescan for 2550 addresses (synced to block 00000000000000000005e9ad3ad1813710a3a71e8728585e112b7d4b8fc28a6d, height 535702)
2018-08-08 00:51:02.634 [INF] LNWL: Removed conflicting tx: (*wire.MsgTx)(0xc420140900)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=1 cap=1) {
  (*wire.TxIn)(0xc420296b40)({
   PreviousOutPoint: (wire.OutPoint) 258bfb43850220242ce87bc030f6cb08ba23a8267e715706884e2eebe08f4f2f:0,
   SignatureScript: ([]uint8) {
   },
   Witness: (wire.TxWitness) (len=4 cap=4) {
    ([]uint8) {
    },
    ([]uint8) (len=72 cap=72) {
     00000000  30 45 02 21 00 eb 77 92  db 58 59 a1 67 6e fe 79  |0E.!..w..XY.gn.y|
     00000010  be 87 b7 0c b3 5d ec 0b  62 07 e1 a6 5e 38 6c 78  |.....]..b...^8lx|
     00000020  34 9d eb d3 6b 02 20 0a  7e 23 dd 6b 74 dc d5 0c  |4...k. .~#.kt...|
     00000030  32 b8 86 dc 69 50 15 ac  5b 5f c5 87 57 c6 30 bf  |2...iP..[_..W.0.|
     00000040  e7 32 b1 b7 39 2f bf 01                           |.2..9/..|
    },
    ([]uint8) (len=72 cap=72) {
     00000000  30 45 02 21 00 f4 89 6e  ba 6e f8 3d c2 51 a6 0a  |0E.!...n.n.=.Q..|
     00000010  71 90 8d 8b 4f 43 16 6b  fc 23 70 93 99 02 a5 7b  |q...OC.k.#p....{|
     00000020  9c f0 45 44 e9 02 20 62  cf 1c f4 9e 30 48 b7 ca  |..ED.. b....0H..|
     00000030  b0 bd 6f 37 ce ca e5 c2  0e a5 3c e3 d2 a2 b4 2a  |..o7......<....*|
     00000040  4d e7 97 db f7 1e 20 01                           |M..... .|
    },
    ([]uint8) (len=71 cap=71) {
     00000000  52 21 03 46 9b 46 19 54  2a 67 0c ba a6 a0 36 39  |R!.F.F.T*g....69|
     00000010  b0 6f 22 fc 52 d2 4e 86  19 5c 5e f1 4f 74 86 d5  |.o".R.N..\^.Ot..|
     00000020  cd 77 e0 21 03 a4 51 1f  c3 34 92 9b 49 e9 b7 21  |.w.!..Q..4..I..!|
     00000030  6e 98 c6 35 cd e7 34 e6  97 cb e1 56 5c 67 7a 59  |n..5..4....V\gzY|
     00000040  b7 3e 93 ea 96 52 ae                              |.>...R.|
    }
   },
   Sequence: (uint32) 2153247315
  })
 },
 TxOut: ([]*wire.TxOut) (len=1 cap=1) {
  (*wire.TxOut)(0xc42058aba0)({
   Value: (int64) 48523,
   PkScript: ([]uint8) (len=22 cap=22) {
    00000000  00 14 b1 05 93 70 30 d5  6d 79 af 69 b9 ae f8 52  |.....p0.my.i...R|
    00000010  42 79 ec 99 2c 61                                 |By..,a|
   }
  })
 },
 LockTime: (uint32) 549852595
})

2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=1)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=2)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=3)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=4)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=5)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=6)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=7)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=8)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=9)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=10)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=11)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=12)}
2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=13)}
2018-08-08 00:51:03.716 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=14)}
2018-08-08 00:51:03.716 [INF] HSWC: Payment circuits loaded: num_pending=21375, num_open=167
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=521038:1472:0, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=531682:1285:0, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=520144:1560:0, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=529484:1779:0, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=519768:1525:0, start_htlc_id=172
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=520346:2274:0, start_htlc_id=601
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=514733:1220:1, start_htlc_id=1338
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=533217:1274:1, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=521416:473:1, start_htlc_id=894
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=532879:833:1, start_htlc_id=14
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=532879:839:1, start_htlc_id=85
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=527755:733:1, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=518528:1022:0, start_htlc_id=0
2018-08-08 00:51:03.783 [INF] HSWC: Trimming open circuits for chan_id=519048:1516:0, start_htlc_id=4
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=532210:1582:1, start_htlc_id=59
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=535547:1037:1, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=532286:1942:1, start_htlc_id=165
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=530245:303:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=532997:1044:1, start_htlc_id=20
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=520745:490:1, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=529095:103:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=532130:2121:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=519619:1375:1, start_htlc_id=815
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=521007:863:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=528983:977:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=519775:2039:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=515675:1474:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=529754:1325:0, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=516403:1187:0, start_htlc_id=494
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=519250:933:1, start_htlc_id=0
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=524757:1305:0, start_htlc_id=194
2018-08-08 00:51:03.784 [INF] HSWC: Trimming open circuits for chan_id=529310:524:1, start_htlc_id=24
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=532880:1608:1, start_htlc_id=22
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=529537:1585:1, start_htlc_id=0
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=524639:109:0, start_htlc_id=0
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=520682:799:1, start_htlc_id=0
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=530773:2352:0, start_htlc_id=4
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=529255:2118:1, start_htlc_id=5
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=519605:1962:1, start_htlc_id=9
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=518451:1704:1, start_htlc_id=268
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=532879:835:1, start_htlc_id=46
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=529353:1209:0, start_htlc_id=26
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=533416:1549:0, start_htlc_id=0
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=519262:586:0, start_htlc_id=10
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=520045:945:0, start_htlc_id=147
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=530911:2042:0, start_htlc_id=0
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=519599:1789:1, start_htlc_id=0
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=532879:836:1, start_htlc_id=24
2018-08-08 00:51:03.785 [INF] HSWC: Trimming open circuits for chan_id=530264:990:0, start_htlc_id=1250
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=530628:680:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=528944:151:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=527697:239:0, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=528093:1521:1, start_htlc_id=107
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=519923:2165:1, start_htlc_id=4
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=532623:1141:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=532212:1117:1, start_htlc_id=1
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=528015:802:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=533680:681:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=517234:2226:1, start_htlc_id=501
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=522109:2045:0, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=521411:1295:1, start_htlc_id=165
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=532335:1691:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=533364:1363:0, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=529124:98:1, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=528060:2228:0, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=528261:1318:0, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=528437:1686:1, start_htlc_id=151
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=530289:2895:0, start_htlc_id=0
2018-08-08 00:51:03.786 [INF] HSWC: Trimming open circuits for chan_id=533364:1039:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=530216:1873:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=535270:342:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=518219:565:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=532879:834:1, start_htlc_id=12
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=519070:1474:1, start_htlc_id=192
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=515078:377:1, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=533372:2216:1, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=520502:1967:0, start_htlc_id=571
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=518510:1692:0, start_htlc_id=172
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=532729:2354:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=533224:606:1, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=514614:1609:0, start_htlc_id=28
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=530867:913:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=529267:572:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=528437:1688:0, start_htlc_id=324
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=535233:866:0, start_htlc_id=0
2018-08-08 00:51:03.787 [INF] HSWC: Trimming open circuits for chan_id=533310:655:1, start_htlc_id=0
2018-08-08 00:51:03.822 [INF] RPCS: RPC server listening on 10.0.0.4:8735
2018-08-08 00:51:03.822 [INF] RPCS: gRPC proxy started at 10.0.0.4:8080
2018-08-08 00:51:03.823 [INF] LTND: Waiting for chain backend to finish sync, start_height=535702
2018-08-08 00:51:03.826 [INF] LTND: Chain backend is fully synced (end_height=535702)!
2018-08-08 00:51:03.827 [INF] LNWL: Started listening for blocks via ZMQ on tcp://127.0.0.1:28332
2018-08-08 00:51:03.836 [INF] HSWC: Starting HTLC Switch
2018-08-08 00:51:03.837 [INF] NTFN: New block epoch subscription
2018-08-08 00:51:03.837 [INF] NTFN: New block epoch subscription
2018-08-08 00:51:04.000 [DBG] HSWC: Closed completed FAIL circuit for 30e90e25f32660a3dcc6bd4310a95511aa32cb39dcfd2b4e3d8c3e5c2c5b3f4a: (519300:1804:0, 107) <-> (514733:1220:1, 102)
2018-08-08 00:51:04.033 [DBG] HSWC: Closed completed FAIL circuit for 1bd292e027916f24d472fdd7105ac1ffff9c6917e463e828b6f3bcc67da7a793: (522514:83:1, 4) <-> (521416:473:1, 277)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for 6db37fb10cfacde416332035a93e65504ee31517e7c1d111567300c31a074a9a: (520557:1036:1, 153) <-> (521416:473:1, 560)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for 6db37fb10cfacde416332035a93e65504ee31517e7c1d111567300c31a074a9a: (520557:1036:1, 154) <-> (521416:473:1, 561)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for 229609274af11243525bd36d67accafea25eab226d32ae2f890073e8b339bb1c: (525368:219:0, 6) <-> (521416:473:1, 674)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for 229609274af11243525bd36d67accafea25eab226d32ae2f890073e8b339bb1c: (525368:219:0, 35) <-> (521416:473:1, 675)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for 229609274af11243525bd36d67accafea25eab226d32ae2f890073e8b339bb1c: (525368:219:0, 54) <-> (521416:473:1, 676)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for 995be7f39fd5d9b388217bcb8d6d3c89daf3f10681eb342490a587cd243aeded: (525368:219:0, 68) <-> (521416:473:1, 677)
2018-08-08 00:51:04.034 [DBG] HSWC: Closed completed FAIL circuit for c339a21b0b03489aa618398bafdfb7028a3fe771098524a0d0a46443b5fc18d0: (525368:219:0, 105) <-> (521416:473:1, 678)
2018-08-08 00:51:04.035 [DBG] HSWC: Closed completed FAIL circuit for d765686b2cb46138f43f1a5147b52b9c207751a759e66f5ee2b7eb4b92692a22: (525368:219:0, 134) <-> (521416:473:1, 679)
2018-08-08 00:51:04.035 [DBG] HSWC: Closed completed FAIL circuit for 15d5c29d6c0451dbdb16b557ba178e3db204019908c3d78c088dd00fe841e855: (525368:219:0, 168) <-> (521416:473:1, 680)
2018-08-08 00:51:04.035 [DBG] HSWC: Closed completed FAIL circuit for 73a28a0ada2b9d4b311b2c14921dec7a5397d1aa44c7dc30ccfca2d0f1fb84a3: (525368:219:0, 195) <-> (521416:473:1, 681)
2018-08-08 00:51:04.035 [DBG] HSWC: Closed completed FAIL circuit for 4b542be022235527bcbc375407a8d68bdc95132bd33091ab2ba8185b32d8097e: (525368:219:0, 216) <-> (521416:473:1, 682)
2018-08-08 00:51:04.035 [DBG] HSWC: Closed completed FAIL circuit for da9f67129c4d06f94811469385e0802b56e2031192db8f119e1709b9d54077b9: (525368:219:0, 230) <-> (521416:473:1, 683)
2018-08-08 00:51:04.035 [DBG] HSWC: Closed completed FAIL circuit for 8db11de486d3b5ba8dd1dd02674da5b43f399f0f206b4259ca17367dcd5a3900: (525368:219:0, 254) <-> (521416:473:1, 684)
2018-08-08 00:51:04.036 [DBG] HSWC: Closed completed FAIL circuit for 8ef2782a0a878ef7ced31e5f8e5aed2416c53abb18db2fd56f9cd0092648af01: (525368:219:0, 268) <-> (521416:473:1, 685)
2018-08-08 00:51:04.036 [DBG] HSWC: Closed completed FAIL circuit for bb1711840d03c4c498af1ab2279277e5daf2052629674cb1c274fb727b5b289e: (525368:219:0, 325) <-> (521416:473:1, 686)
2018-08-08 00:51:04.036 [DBG] HSWC: Closed completed FAIL circuit for edc7719c703a7221f48e7c3a27e1faab44221aa2243e477e44988b2a4d4c8783: (0:0:0, 52760) <-> (521416:473:1, 748)
2018-08-08 00:51:04.036 [DBG] HSWC: Closed completed FAIL circuit for c4e98cb8c3d7977d4b03c28fe093687b61a45564a47fdd96e8d884f3762b18bc: (0:0:0, 52788) <-> (521416:473:1, 752)
2018-08-08 00:51:04.036 [DBG] HSWC: Closed completed FAIL circuit for c4e98cb8c3d7977d4b03c28fe093687b61a45564a47fdd96e8d884f3762b18bc: (0:0:0, 52818) <-> (521416:473:1, 756)
2018-08-08 00:51:04.036 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Cannot find pending payment with ID 52760
2018-08-08 00:51:04.036 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Cannot find pending payment with ID 52788
2018-08-08 00:51:04.037 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Cannot find pending payment with ID 52818
2018-08-08 00:51:04.101 [DBG] HSWC: Closed completed FAIL circuit for 3608f39921dcd4d6170b2b46eb179cf10b2682abbb62b2c9df5a8b502c393725: (527114:1480:1, 525) <-> (532210:1582:1, 2)
2018-08-08 00:51:04.144 [DBG] HSWC: Closed completed FAIL circuit for 4689a97abb7f99a75efed862b3ffffffffffffffffffffffffffffffffffffff: (532176:2104:0, 30) <-> (532997:1044:1, 5)
2018-08-08 00:51:04.322 [DBG] HSWC: Closed completed FAIL circuit for fdbbe08449102778d9bd5661f752151e9fd5105a57bace30a08c320579eb2e6d: (514733:1220:1, 0) <-> (516403:1187:0, 452)
2018-08-08 00:51:04.346 [DBG] HSWC: Closed completed FAIL circuit for 0f099992a4c52917cec8c38cbba562e3f6e1e5dacdeacd5672ae1336c4d75649: (517234:2226:1, 61) <-> (524757:1305:0, 52)
2018-08-08 00:51:04.346 [DBG] HSWC: Closed completed FAIL circuit for 22df70af9484557b7c62b5b697bac78a2d420235612b940eb51aeb33422444da: (517234:2226:1, 67) <-> (524757:1305:0, 53)
2018-08-08 00:51:04.346 [DBG] HSWC: Closed completed FAIL circuit for 4689a97abb7f99a75efed862b3ffffffffffffffffffffff7bf712a3c8494ec2: (532286:1942:1, 12) <-> (524757:1305:0, 192)
2018-08-08 00:51:04.476 [DBG] HSWC: Closed completed FAIL circuit for df2c0dc5345ba9a63049ae9491fa2c97cb5309c25f4efdf0bcf4bb5fb84f69e3: (529254:753:1, 32) <-> (529255:2118:1, 1)
2018-08-08 00:51:04.477 [DBG] HSWC: Closed completed FAIL circuit for df2c0dc5345ba9a63049ae9491fa2c97cb5309c25f4efdf0bcf4bb5fb84f69e3: (519435:1016:1, 22) <-> (529255:2118:1, 2)
2018-08-08 00:51:04.486 [DBG] HSWC: Closed completed FAIL circuit for 92646e774c547e9e6d0540e2c4098da45203d088c61590d36ed123136d5a140e: (518774:767:0, 169) <-> (519605:1962:1, 7)
2018-08-08 00:51:04.496 [DBG] HSWC: Closed completed FAIL circuit for a7e2c6bfcf963e23745562f796167f690074ad39b16785e47a64e1119cb1369e: (521508:2091:1, 11) <-> (518451:1704:1, 0)
2018-08-08 00:51:04.496 [DBG] HSWC: Closed completed FAIL circuit for 04c42155779abbec2dc982bd0462a83c9b8fa02557782a778c987fd9d643ac37: (518642:1660:1, 2) <-> (518451:1704:1, 148)
2018-08-08 00:51:04.496 [DBG] HSWC: Closed completed FAIL circuit for 6acb6b4bef9bb3f5ac18bb852592f810f728881516b19bfc72db3a83d75689c0: (518642:1660:1, 4) <-> (518451:1704:1, 149)
2018-08-08 00:51:04.496 [DBG] HSWC: Closed completed FAIL circuit for 5f8b5ed4ef50cef200cb629aa6e20fdc07a782c80a45bb0deefc9b44a4dce674: (529254:753:1, 35) <-> (518451:1704:1, 263)
2018-08-08 00:51:04.496 [DBG] HSWC: Closed completed FAIL circuit for 17b749b87654761fa737b27759b604bfa6426ee81d8f14e17aa7d4a3facc6099: (527114:1480:1, 437) <-> (518451:1704:1, 264)
2018-08-08 00:51:04.520 [DBG] HSWC: Closed completed FAIL circuit for ae71d348a0ce8c166b5409e413bcd58e09afbcdd5bd1a92df5eb3b8631dece84: (518642:1660:1, 96) <-> (529353:1209:0, 12)
2018-08-08 00:51:04.521 [DBG] HSWC: Closed completed FAIL circuit for ae71d348a0ce8c166b5409e413bcd58e09afbcdd5bd1a92df5eb3b8631dece84: (518642:1660:1, 97) <-> (529353:1209:0, 13)
2018-08-08 00:51:04.521 [DBG] HSWC: Closed completed FAIL circuit for 458530fb948d4a4150f798f0669489df70bc9cbb24e4a6bf293a654753b71aad: (527383:1306:1, 51) <-> (529353:1209:0, 16)
2018-08-08 00:51:04.521 [DBG] HSWC: Closed completed FAIL circuit for 92c1ea93aa89e0bf6a44448c6bf31dde32b0062b615736cc1924d94427a1fd68: (527383:1306:1, 55) <-> (529353:1209:0, 17)
2018-08-08 00:51:04.521 [DBG] HSWC: Closed completed FAIL circuit for c7ffceac5fb6dfa2963ef2b6d5c2e0f924c1dacf2d60a22ad49943151c53879f: (529254:750:1, 3) <-> (529353:1209:0, 18)
2018-08-08 00:51:04.972 [DBG] HSWC: Closed completed FAIL circuit for b56f62fee02d46c043aace2164831c5deb540a9e2e3994e5751e8366f1e71f6e: (527114:1480:1, 473) <-> (528437:1688:0, 91)
2018-08-08 00:51:04.972 [DBG] HSWC: Closed completed FAIL circuit for b56f62fee02d46c043aace2164831c5deb540a9e2e3994e5751e8366f1e71f6e: (527114:1480:1, 474) <-> (528437:1688:0, 92)
2018-08-08 00:51:04.973 [DBG] HSWC: Closed completed FAIL circuit for 06c2a6be476d87ce215799af925ce5178f40d0421373dd2c7883f52ec2ebec48: (528437:1686:1, 17) <-> (528437:1688:0, 93)
2018-08-08 00:51:04.973 [DBG] HSWC: Closed completed FAIL circuit for 0c5abac34e06fcc5cb13397b29a69ac630038aea63cd4fa75ba847fb92a7e4fa: (528437:1686:1, 18) <-> (528437:1688:0, 94)
2018-08-08 00:51:04.973 [DBG] HSWC: Closed completed FAIL circuit for 300c8cffcbe882ac0eb4929ec865e4de91d4dd6dda85b825dac370474b0cae40: (528437:1686:1, 19) <-> (528437:1688:0, 95)
2018-08-08 00:51:04.973 [DBG] HSWC: Closed completed FAIL circuit for 300c8cffcbe882ac0eb4929ec865e4de91d4dd6dda85b825dac370474b0cae40: (528437:1686:1, 20) <-> (528437:1688:0, 96)
2018-08-08 00:51:04.993 [INF] NTFN: New block epoch subscription
2018-08-08 00:51:05.017 [INF] UTXN: Re-registering confirmations for 1 already graduated heights below height=535702
2018-08-08 00:51:05.017 [INF] UTXN: Re-registering confirmation for kindergarten sweep transaction at height=527663
2018-08-08 00:51:05.017 [INF] UTXN: Sweeping 1 CSV-delayed outputs with sweep tx (txid=c710183ae536506651ed2670da8cc72ae153e61197f67f2ebb937401c74e0ad7): (*wire.MsgTx)(0xc4200bcfc0)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=1 cap=1) {
  (*wire.TxIn)(0xc420707980)({
   PreviousOutPoint: (wire.OutPoint) 5e2d590fcc48098ff6fe0dee6d6a2f624190fc80f0fd7964396c4e08c2526bf1:0,
   SignatureScript: ([]uint8) {
   },
   Witness: (wire.TxWitness) (len=3 cap=3) {
    ([]uint8) (len=72 cap=72) {
     00000000  30 45 02 21 00 ce 70 d3  73 b7 db da a3 2e 5c bd  |0E.!..p.s.....\.|
     00000010  54 7e f7 d9 a1 87 ee ef  bb 2a 5f 45 57 39 e5 db  |T~.......*_EW9..|
     00000020  58 07 47 af 8c 02 20 0b  c0 78 74 b8 8a 13 ba 05  |X.G... ..xt.....|
     00000030  18 06 f7 df f2 fe 86 a9  ab 3d 44 98 b2 a1 ab 93  |.........=D.....|
     00000040  50 79 8c e2 91 7c 89 01                           |Py...|..|
    },
    ([]uint8) {
    },
    ([]uint8) (len=77 cap=77) {
     00000000  63 21 03 b8 4d 01 38 a7  7f 7f 4c 67 a6 75 de aa  |c!..M.8...Lg.u..|
     00000010  af c2 e4 b6 7e e0 ee 41  0e 4a 9c 4e e9 26 1c bd  |....~..A.J.N.&..|
     00000020  05 e1 ab 67 02 90 00 b2  75 21 02 1d fb 7d 8a 1c  |...g....u!...}..|
     00000030  e5 09 40 9e 5b ec 0b a3  69 5c c0 39 6c 8c 9a 64  |..@.[...i\.9l..d|
     00000040  42 8d 01 58 45 e4 a9 b5  8f 81 9b 68 ac           |B..XE......h.|
    }
   },
   Sequence: (uint32) 144
  })
 },
 TxOut: ([]*wire.TxOut) (len=1 cap=1) {
  (*wire.TxOut)(0xc420dfa2e0)({
   Value: (int64) 241,
   PkScript: ([]uint8) (len=22 cap=22) {
    00000000  00 14 b4 d8 ab 62 20 64  d2 41 55 38 36 25 6c 04  |.....b d.AU86%l.|
    00000010  62 56 7b 77 64 2c                                 |bV{wd,|
   }
  })
 },
 LockTime: (uint32) 0
})

2018-08-08 00:51:05.028 [ERR] UTXN: unable to broadcast sweep tx: -26: 64: dust, (*wire.MsgTx)(0xc4200bcfc0)({
 Version: (int32) 2,
 TxIn: ([]*wire.TxIn) (len=1 cap=1) {
  (*wire.TxIn)(0xc420707980)({
   PreviousOutPoint: (wire.OutPoint) 5e2d590fcc48098ff6fe0dee6d6a2f624190fc80f0fd7964396c4e08c2526bf1:0,
   SignatureScript: ([]uint8) {
   },
   Witness: (wire.TxWitness) (len=3 cap=3) {
    ([]uint8) (len=72 cap=72) {
     00000000  30 45 02 21 00 ce 70 d3  73 b7 db da a3 2e 5c bd  |0E.!..p.s.....\.|
     00000010  54 7e f7 d9 a1 87 ee ef  bb 2a 5f 45 57 39 e5 db  |T~.......*_EW9..|
     00000020  58 07 47 af 8c 02 20 0b  c0 78 74 b8 8a 13 ba 05  |X.G... ..xt.....|
     00000030  18 06 f7 df f2 fe 86 a9  ab 3d 44 98 b2 a1 ab 93  |.........=D.....|
     00000040  50 79 8c e2 91 7c 89 01                           |Py...|..|
    },
    ([]uint8) {
    },
    ([]uint8) (len=77 cap=77) {
     00000000  63 21 03 b8 4d 01 38 a7  7f 7f 4c 67 a6 75 de aa  |c!..M.8...Lg.u..|
     00000010  af c2 e4 b6 7e e0 ee 41  0e 4a 9c 4e e9 26 1c bd  |....~..A.J.N.&..|
     00000020  05 e1 ab 67 02 90 00 b2  75 21 02 1d fb 7d 8a 1c  |...g....u!...}..|
     00000030  e5 09 40 9e 5b ec 0b a3  69 5c c0 39 6c 8c 9a 64  |..@.[...i\.9l..d|
     00000040  42 8d 01 58 45 e4 a9 b5  8f 81 9b 68 ac           |B..XE......h.|
    }
   },
   Sequence: (uint32) 144
  })
 },
 TxOut: ([]*wire.TxOut) (len=1 cap=1) {
  (*wire.TxOut)(0xc420dfa2e0)({
   Value: (int64) 241,
   PkScript: ([]uint8) (len=22 cap=22) {
    00000000  00 14 b4 d8 ab 62 20 64  d2 41 55 38 36 25 6c 04  |.....b d.AU86%l.|
    00000010  62 56 7b 77 64 2c                                 |bV{wd,|
   }
  })
 },
 LockTime: (uint32) 0
})

2018-08-08 00:51:05.028 [ERR] UTXN: Failed to re-register for kindergarten sweep transaction at height=527663: -26: 64: dust
2018-08-08 00:51:05.028 [ERR] UTXN: Failed to regraduate outputs at height=527663: -26: 64: dust
2018-08-08 00:51:05.226 [INF] CNCT: Creating ChannelArbitrators for 87 active channels

cfromknecht commented 6 years ago

Those logs look like they're from the old patch?

2018-08-08 00:51:03.715 [INF] HSWC: Ignoring stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=12)}

should be

2018-08-08 00:51:03.715 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=12)}

cfromknecht commented 6 years ago

Also that channel looks like it was unilaterally closed, not cooperatively closed. Either way the channel only had 733 satoshis in it, so there's not much we can do to sweep it atm.

Horndev commented 6 years ago

@cfromknecht Doh - you're right. I forgot to copy the binaries after uploading to server.

here is the updated log with the actual patch:

2018-08-08 01:07:57.655 [INF] HSWC: Restoring in-memory circuit state from disk
2018-08-08 01:07:57.715 [INF] LNWL: Started rescan from block 000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4 (height 535704) for 2551 addresses
2018-08-08 01:07:57.890 [INF] LNWL: Starting rescan from block 000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4
2018-08-08 01:07:57.893 [INF] LNWL: Rescan finished at 535704 (000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4)
2018-08-08 01:07:57.893 [INF] LNWL: Catching up block hashes to height 535704, this might take a while
2018-08-08 01:07:57.902 [INF] LNWL: Done catching up block hashes
2018-08-08 01:07:57.902 [INF] LNWL: Finished rescan for 2551 addresses (synced to block 000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4, height 535704)
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=1)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=2)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=3)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=4)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=5)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=6)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=7)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=8)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=9)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=10)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=11)}
2018-08-08 01:07:58.924 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=12)}
2018-08-08 01:07:58.925 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=13)}
2018-08-08 01:07:58.925 [INF] HSWC: Removing stray keystone: {(Chan ID=0:0:0, HTLC ID=63771) (Chan ID=0:0:0, HTLC ID=14)}
2018-08-08 01:07:58.934 [INF] HSWC: Payment circuits loaded: num_pending=21381, num_open=167

Node went up and running no problem.

I shut it down and ran again. The issue was not in the log the next time.

2018-08-08 01:10:41.069 [INF] HSWC: Restoring in-memory circuit state from disk
2018-08-08 01:10:41.528 [INF] LNWL: Started rescan from block 000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4 (height 535704) for 2552 addresses
2018-08-08 01:10:41.694 [INF] LNWL: Starting rescan from block 000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4
2018-08-08 01:10:41.696 [INF] LNWL: Rescan finished at 535704 (000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4)
2018-08-08 01:10:41.696 [INF] LNWL: Catching up block hashes to height 535704, this might take a while
2018-08-08 01:10:41.706 [INF] LNWL: Done catching up block hashes
2018-08-08 01:10:41.706 [INF] LNWL: Finished rescan for 2552 addresses (synced to block 000000000000000000031c4ff80ac7a95a5488a655673c9a1ee8aaa0e9bbbaf4, height 535704)
2018-08-08 01:10:41.988 [INF] HSWC: Payment circuits loaded: num_pending=21381, num_open=167

cfromknecht commented 6 years ago

@Horndev awesome! It should be safe to revert to master, the stray keystones should no longer be an issue :)

I'll add some similar logic to handle this case in #1698, so that anyone else upgrading from an older commit doesn't experience this.

cfromknecht commented 6 years ago

fwiw, @vapopov is working on a solution to coalesce and sweep dust outputs in #1184

lightningnetwork / lnd