lnd neutrino sync time

[kilrau]

Currently we are looking at 15 minutes sync time increase per week (25k blocks) for lndbtc. Similar for lndltc. We should explore automatically (daily?) adding checkpoints and make neutrino only sync from there to keep sync time below the target of 5 minutes.

[kilrau]

Actually most of us are seeing much longer sync times for the current 22k blocks on btc/ltc. We probably have to dive into that rather sooner than later.

[kilrau]

So for some of us, lnd was doing this (and that's why the sync takes ages):

2020-04-24 17:25:42.415 [INF] CRTR: Block 007872d989185ea29032d54f654e90b750f6496e9eb4f00f290db4f417f266ac (height=7772) closed 0 channels
2020-04-24 17:25:42.664 [DBG] BTCN: Received cfilter from 35.231.222.142:38555 (outbound)
2020-04-24 17:25:42.664 [INF] CRTR: Pruning channel graph using block 65cd496a16fe855e0d7e399d184901cf6ecef49925c946f8ce5acc02341ccf73 (height=7773)
2020-04-24 17:25:42.664 [DBG] BTCN: Fetching filters for heights=[7774, 7774], stophash=0239775ebf2c8705d25e986358ecc334d12a870e8422b3576d0f3de532d8f147
2020-04-24 17:25:42.665 [DBG] BTCN: Sending getcfilters to 35.231.222.142:38555 (outbound)
2020-04-24 17:25:42.666 [INF] CRTR: Block 65cd496a16fe855e0d7e399d184901cf6ecef49925c946f8ce5acc02341ccf73 (height=7773) closed 0 channels
2020-04-24 17:25:43.060 [DBG] BTCN: Received cfilter from 35.231.222.142:38555 (outbound)
2020-04-24 17:25:43.061 [DBG] BTCN: Fetching filters for heights=[7775, 7775], stophash=1f30c442f2259cdc4f5ddc8e39e4375de853edc24cf1ac58711aaa8c348f9e84
2020-04-24 17:25:43.061 [INF] CRTR: Pruning channel graph using block 0239775ebf2c8705d25e986358ecc334d12a870e8422b3576d0f3de532d8f147 (height=7774)
2020-04-24 17:25:43.061 [DBG] BTCN: Sending getcfilters to 35.231.222.142:38555 (outbound)
2020-04-24 17:25:43.063 [INF] CRTR: Block 0239775ebf2c8705d25e986358ecc334d12a870e8422b3576d0f3de532d8f147 (height=7774) closed 0 channels
2020-04-24 17:25:43.414 [DBG] BTCN: Received cfilter from 35.231.222.142:38555 (outbound)

[raladev]

Logs of nodes with 10 min sync time: logs2.zip After that I got 1 Active ltc channel and 1 inactive btc channel.

[kilrau]

@reliveyy :

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
1da26d8fb14f        simnet_xud_1        0.49%               112.1MiB / 15.58GiB   0.70%               11.8MB / 11.9MB     0B / 39.7MB         14
45ebb703fc68        simnet_connext_1    0.03%               116.3MiB / 15.58GiB   0.73%               2.73MB / 3.62MB     32.8kB / 139kB      35
2114462c999a        simnet_lndltc_1     0.03%               311.2MiB / 15.58GiB   1.95%               14MB / 3.83MB       0B / 16.5GB         26
51762178a9c7        simnet_lndbtc_1     0.06%               326.2MiB / 15.58GiB   2.05%               16.2MB / 7.23MB     385kB / 1.57GB      25
13dc5fc0d540        simnet_utils_1      0.01%               25.5MiB / 15.58GiB    0.16%               218kB / 70.8kB      0B / 0B             3

For me lndltc writes to disk more than 16GB

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
97acea2e5f48        simnet_utils_1      0.01%               25.46MiB / 15.39GiB   0.16%               212kB / 69.7kB      24.6kB / 16.4kB     3
6e0502085da3        simnet_xud_1        0.76%               117.2MiB / 15.39GiB   0.74%               12.4MB / 12.3MB     12.3kB / 2.84MB     14
1f5f3b2607b3        simnet_connext_1    0.99%               116.7MiB / 15.39GiB   0.74%               2.81MB / 3.74MB     0B / 115kB          35
194ebfe1c696        simnet_lndltc_1     0.10%               303.2MiB / 15.39GiB   1.92%               13.1MB / 3.67MB     406kB / 2.38GB      25
fe3a7ef4e3c1        simnet_lndbtc_1     0.11%               302.8MiB / 15.39GiB   1.92%               12.9MB / 3.78MB     12.3kB / 2.37GB     25

for me (@kilrau ) it's 2.x GB each.

That's a good hint. If LND is downloading 16GB of data via tor and then pruning then it makes sense that it takes a while.

[sangaman]

So for some of us, lnd was doing this (and that's why the sync takes ages):

I had that, and then I also am going through all the ~23k blocks with this message:

8:07:52.884 [DBG] CNCT: ChannelArbitrator(ecf12ea1f2430bfeadcb3cbdbd9e9b98a92ff827080319181a56577e9268c7f7:0): checking commit chain actions at height=20201, in_htlc_count=0, out_htlc_count=0
2020-04-24 18:07:52.885 [INF] UTXN: Attempting to graduate height=20201: num_kids=0, num_babies=0

It's been roughly 2 hours so far.

[reliveyy]

The lnd blkio output over 1GB is mainly due to --debuglevel=debug launching option. I was trying to use strace to monitor write system call of lnd process and I found it writen to lnd.log mostly. But after I set --debugleve=error the blkio output of lnd is still ~300M which is still higher than other nodes. @kilrau

[reliveyy]

It took 2:06:06 to get channels for lndbtc and lndltc. And it will take more than 2 hours to be fully synced for sure. Here is part of pytest -s tests/integration/test_simnet.py output in project xud-docker:

lndbtc syncing: 26497/26781
lndbtc status: lnd-BTC has no active channels
lndbtc syncing: 26554/26781
lndbtc status: lnd-BTC has no active channels
lndbtc syncing: 26613/26781
lndbtc status: lnd-BTC has no active channels
lndbtc syncing: 26667/26781
lndbtc status: Ready
Try to get lndltc block height (retry=0)
lndltc block height: 26859
lndltc syncing: 26648/26859
lndltc status: lnd-LTC has no active channels
lndltc syncing: 26705/26859
lndltc status: lnd-LTC has no active channels
lndltc syncing: 26795/26859
lndltc status: Ready

test-simnet.log

[reliveyy]

@kilrau I've test it on GCP with a new VM instance. The simnet test case finished in 0:14:48 which is your case within 15 minutes. So I believe the syncing time is highly related to the network condition. Our xud1.simnet.exchangeunion.com is in us-central1-a which probably won't have good network connection to everywhere in the world. In order to provide faster syncing time, I would sugguest to add more simnet nodes hosted by us around the world.