[Testing / Research] MPP, path finding, payment reliability test

Description

This is just a research post to follow up all the testing in regards of failed payments with different apps and routes

I open up this discussion to find out what is going on with LN payments made with Blixt and over LN in general. Some payments work, some don't. I put a lot of work into this, testing and trying to understand why it fails and not always. Please add your testing scenarios here and please add all details you can (hardware, software, steps done etc) all these are very important to have the right diagnostic. Seems that is not 100% only Blixt fault but also LND code affect these failed payments. So we are trying to find out and how to fix it.

My tests until now

Environment:

Blixt v0.6.1 (official from github), Device A Motorola G9+, 4GB RAM, Android 11, Blixt wallet A
Blixt v0.6.1 (strict-false testing version), Device B Motorola G42, 6GB RAM, Android 12, Blixt wallet B
I have another device C with Android 8.1 but on that one, any Blixt version is stuck in initial sync and scan. So is out of testing for now.
for the sake of good connectivity and eliminate any possible issues with connection, I will NOT use Tor, just clearnet.
each Blixt have only 1 channel to Blixt node (for the moment I wanted to see how is going with one channel)
I have also ready to use other LN wallet apps: OBW, SBW, Phoenix, WoS, Bluewallet, Electrum, LNbits, LNTXBOT, CoinOS, Zeus (connected to a small private CLN node and lndhub accounts)

1st Test - testing various levels of amounts

using 4 levels of invoice amounts: under 10k sats, under 20k, under 50k, under 100k
sending between all these wallet apps back and forth, with random amounts, not rounded
monitoring (if is possible) the number of hops and fees paid
In Blixt before any payment I did the "reset payment routing knowledge".
In Blixt I used MPP activated (due to the fact that I have 1 channel only)
I always wait for blocks sync, channel come online and also checking logs for graph sync, to be sure is all ready.

1. Sending / receiving with Blixt:

payments under 50k worked only to LNbits and Phoenix, all other bigger amounts failed. Also I notice that took at least 20 seconds to finalize the payment.
receiving payments into Blixt worked instantly, except the one of 100k because I didn't have enough inbound liquidity. So I have to do another test with an empty channel to see how it works. But I have the feeling that will not be any problem.

2. Sending / receiving with OBW/SBW:

all payments worked fine, except one of 99 900 sats because of lack of liquidity on my side.
receiving payments worked also fine, except one time with 49950 sats, taking a bit longer and 2 intents. I think that was due to bad connectivity

3. Sending / receiving with Phoenix:

all payments worked just fine. The only thing with Phoenix is that is opening a channel for each incoming payment. later is quite difficult to manage them. so the best way is to close them all, and from onchain using a swap, back to LN in a bigger channel.
receiving was without any problem.

4. Sending / receiving with Bluewallet:

some payments didn't worked at first intent. After waiting few minutes, trying again and worked. Can be due to low liquidity in the path or on their LNDHUB side.
receiving payments was without any problem, any amount

Electrum, LNTXBOT, LNBits, WoS, CoinOS were used as intermediary destinations (also using LNURL, LN address to test various ways to send payments). I couldn't test with Zeus / CLN node, due to the fact that my testing CLN node was fucked and now I am in re-sync. But I used Zeus for these tests with a lndhub account from LNTXBOT and LNbits. All payments had at least 3 hops and no more than 20 sats in fees/tx, sometimes ridiculous small amounts (I was surprised).

Conclusion

Something is going on with payments sent from Blixt. The weird thing is that is not always and with not all amounts. I will try to repeat this test process in few days again.

Please do your tests if you can or just report here, if you do a normal payment and want to report more insight if you have a failed tx.

In general I have observed path finding issues with not just blixt but any neutrino based node that has only one or very few channels such as my 2nd node connected to Zeus wallet that I use for payments. These nodes have a smaller view of the graph because their source for gossip data is much smaller compared to a regular routing node.

I think mobile wallets in general have to rely on external sources for gossip or path finding for two reasons, speed and reliability. I can think of two solutions where we could improve reliability and speed.

Get gossip from else where such as a big routing node:

Pros:

Larger and up to date graph
Faster than syncing graph ourselves

Cons:

We have to look to see if LND can support this.
Small trust issues
Gossip server should have very good uptime.

Out source path finding to a routing node:

Pros:

No graph sync needed at all which means payments could be fast and very reliable

Cons:

Give up on privacy because the routing node would know your payments history

IMO I don't think a mobile wallet can do everything by itself and some of its functions have to be outsourced to an external server that does heavy lifting jobs, hard part is how do we balance out speed & reliability with privacy. It's like using Tor vs Clearnet, each has its own upside.

2nd Test - Sending from Blixt to various destinations

opened a new channel to Kevin node, of 130k sats
MPP option activated always
Blixt v0.6.1 (special testing edition)

1. Payment A - from Blixt to OBW

9150 sats
3 hops
fee 18 sats (expensive!), I wonder who took such fee?
SUCCESS payment was instant done, sent through Kevin channel
I was expecting to use multiple chunks from both channels (using MPP), but was all from one channel

2. Payment B - from Blixt to Phoenix

23520 sats
4 hops
fee 8 sats (wtf?!)
I couldn't scan the QR generated by Phoenix, keep saying payment details are wrong. I then copied the LN invoice code and paste it into Blixt, then worked. I think Phoenix was trying to send more info in that QR than a simple invoice, because was trying to open a new channel. Quite weird how is operating Phoenix the channels.
SUCCESS payment was instant, sent through Blixt channel
I was expecting to use multiple chunks from both channels (using MPP), but was all from one channel

3. Payment C - from Blixt to LNbits.com

59500 sats
3 hops
fee 28 sats
It couldn't scan the QR, I have no idea why is working to scan with smaller amounts but not bigger than 20k, but once I copy/paste the LN invoice code it worked
SUCCESS payment was taking 12 seconds to finish, sent through Kevin channel
I was expecting to use multiple chunks from both channels (using MPP), but was all from one channel

4. Payment D - from Blixt to WoS

85100 sats
liquidity to send: 1 channel with 56500 sats and another one with 97280 sats, so MPP should be triggered in full this time
destination Wallet of Satoshi (I choose this one for the bigger amount just to be sure is not about liquidity)
FAILED payment with "path not found" after 58 sec
Again the QR couldn't be scanned from WoS with Blixt camera. I had to copy/paste the LN invoice code
tried a 2nd intent with "reset payment route" option... "payment time out" after 87 sec

3rd Test - refill back the channel in Blixt

closed the channel with Blixt node
trying to refill the remaining channel with Kevin, up to 130k sats, from different sources where I already sent sats

1. Payment A - to receive in Blixt

67500 sats
from SBW, with 2 HC, expecting to take chunks from both channels
MPP was trying to split it up to 6 shards but after many intents, payment FAILED
tried the same invoice from LNbits.com and payment FAILED

2. Payment B - to receive in Blixt

49950 sats
from LNBits.com
SUCCESS payment, after 16 sec payment was done correctly

3. Payment C - from Blixt to WoS

79850 sats
I couldn't be able to scan the QR code with Blixt camera, had to copy/paste
SUCCESS payment went through in 3 sec, correctly
2 hops with 63 sats fee !!!

4. Payment D - from SBW to Blixt

62100 sats
I had 2 HC, one with 6145 sats and one with 84190 sats
SUCCESS payment went through in 3 sec, taking MPP 2 parts and almost emptying the smaller channel.
fee 29 sats

5. Payment E - from WoS / Phoenix to Blixt

36500 sats invoice
In Blixt I have 36560 sats inbound liquidity able to receive. I want to push to the limit this channel to see if I can fill it top
I was keeping alive Blixt app all the time, just to be sure there are no communication "issues" waiting.
WoS took 10 minutes to timeout - FAILED payment
Phoenix, I have 3 channels (12000, 24500, 20520), payment FAILED after less than 30 sec
Tried a 2nd intent with only 35000 sats
SUCCESS payment from Phoenix, in 3 sec
fee 42 sat !!! (LOL quite expensive)
Remaining inbound space: 1560 sats that wasn't able to receive anymore in this channel

4th Test - adding a dunder channel and refill at maximum

Blixt v0.6.1 build 30/115 (strict-graph vers)
keeping the channel with Kevin
opening a Dunder channel with Kevin. You have to setup in Blixt: use Dunder channels and after that go to "Set Dunder channel" and put this server: https://dunder.eldamar.icu. This is for testing purpose only.
used 79000 sats from WoS to open a Dunder channel with Kevin node. The opening was instant! Deducted fee was quite high (5831sats), due to mempool congestion and Kevin commision. Understandable.
A dunder channel was open after few minutes, with 73k sats as outbound liquidity and the rest until 210k sats as inbound liquidity. Good start to test.

1. Payment A - from SBW to Blixt

25000 sats invoice
In Blixt was 2 channels: one with 1560 sats inbound available and the dunder ch with 129452 sats inbound available
SUCCESS in just 2 sec payment was done, perfectly
fee paid 7 sats

2. Payment B - from LNbits to Blixt

58450 sats invoice
In Blixt are available to receive 104452 sats in Dunder Kevin channel and 1612 sats in normal Kevin channel
FAILED payment after 3 min, path not found
2nd intent to make the payment was SUCCESS after 2min
patience is the key and never give up.

3. Payment C - from Phoenix to Blixt

9000 sats invoice, just a small amount,
I wanted to refill at maximum the remaining inbound balance in my Dunder channel with Kevin, where I have 47k available to receive
FAILED payment, saying that there is no sufficient inbound liquidity. That is not true!
Tried again same payment towards the same Kevin node, but on its LNbits instance wallet and worked from Phoenix. Is unclear why Phoenix is taking weird paths paying to Blixt.
Be aware that Phoenix is randomly going offline if you have multiple channels. And the sats from that offline channel WILL NOT appear on the main balance, not even mentioning that are offline. So if that peer is not going to be online, the only chance is to close all channels in Phoenix and hop you restore the sats in onchain. But this insane UI.

4. Payment D - from LNTXBOT to Blixt

remaining available inbound in Blixt is 40620 sats and 1612 sats in both channels with Kevin
40500 invoice from Blixt

Nice to see that Blixt is showing in real time how the partial amounts are coming and HTLCs pending, during the payment intent. LNTXBOT was trying to send the payment in multiple shards.

FAILED payment in LNTXBOT


LNTXBOT: Payment 865d1154b64d933005b14864e340f56f28f5d262dcc2038dcfd06f497bc35ed8 failed.

TemporaryChannelFailure at 763710x853x1 (028c58)
TemporaryChannelFailure at 763862x2349x1 (028c58)
TemporaryChannelFailure at 763862x2349x1 (028c58)
TemporaryChannelFailure at 763710x853x1 (028c58)
TemporaryChannelFailure at 763710x853x1 (028c58)

5. Payment E - from another Blixt to Blixt

Blixt sender is v0.6.1 build 30/114 (official version), Blixt receptor is v0.6.1 build 30/115 (strict-graph-vers)
same 40500 invoice
FAILED payment, with no route path found

6. Payment F - from LNbits.com to Blixt

1543 sats testing a small amount to see if I can fill the remaining inbound of 1612 sats in one channel.
SUCCESS payment after 8 sec
payment entered into the channel with larger inbound available, even that both channels are with the same peer

7. Payment G - from LNbits.com to Blixt

trying to push again to the limit to fill up the channel
This is the status of the Blixt channel before making the payment

35500 invoice from Blixt
This is a real time status of the Blixt channel, during the payment process

What is interesting is that it shows the commit fee going high with 5 HTLCs, meanwhile the available "can receive" go down to 2248 sats. That means will never be possible to refill a channel at maximum. So that "can receive" is not true!

FAILED payment after 60 sec
after the payment it failed, the commit fee comes back to previous amount that was before the payment intent.

8. Payment H - several pushes from different sources towards Blixt to fill up totally

1st intent with 33500 invoice - SUCCESS payment from Kevin node
2nd intent with smaller amounts from LNbits.com and Kevin node. Were many payments, going smaller and smaller in amount. Luckily also the commitment fees went down, due to emptying the mempool. So the amount "can receive" went up and total commit fee went down.
In the end both channels were filled up at maximum possible, as following:

Mentions:

The actual way how numbers are displayed for each channel details are wrong, will be necessary a redesign of formula to display the correct calculations for each section.
"Can Receive" is actually what is displayed now - "local reserve"
The 1st channel was a Dunder channel, initiated by Kevin. So there's a 540 sats (2640 - 2100) that couldn't be sent anymore, must be a reserve on his side or something.
The 2nd channel was a normal channel open from my Blixt towards Kevin node. That channel in the end we fill it up using invoices of 1 sat until the last one, when was displayed the "can receive" equal to "local reserve".

Test Blixt wallet path finding capabilities number 1.

Environment.

Phones used; Redmi Note 9 with Blixt Wallet (V0.6.0-No-strict-graph) Samsung Note 10+ 5G with Blixt Wallet (V0.6.0-No-strict-graph) Bluestacks emulator(Pixel 6) with Blixt Wallet (V0.6.0-No-strict-graph) & Blixt Wallet (V0.6.1) Realme C11 2021 with Blixt Wallet (V0.6.1)

Blixt versions used;

Blixt Wallet (V0.6.0-No-strict-graph) Blixt Wallet (V0.6.1)

-MPP was set to 16 parts max (whatever setting that is in Blixt on or off) -Single channel in each Blixt wallet

Payment A.

Blixt Wallet (V0.6.0-No-strict-graph) to Blixt wallet Blixt Wallet (V0.6.0-No-strict-graph) Near instant Channels to the same Parent node Tested up to 2M sats with 100% successes

Payment B.

Blixt Wallet (V0.6.0-No-strict-graph) to Blixt wallet Blixt Wallet (V0.6.0-No-strict-graph) Near instant <2 hop to destination Tested up to 2M sats with 100% success China phone or not payments still work

Payment C.

Blixt Wallet (V0.6.0-No-strict-graph) to Breez (latest release) Near instant Same parent node Tested up to 1M sats with 100% success

Payment D.

Blixt Wallet (V0.6.0-No-strict-graph) to WoS (latest release) Within 25-30 seconds <4 hops Tested up to 100k sats with 100%. china phones again are slower. lol

Payment E.

Blixt Wallet (V0.6.0-No-strict-graph) to OBW (V0.1.6) Near instant <2 Hops Tested up to 200k sats with 100%

Payment F.

Blixt Wallet (V0.6.0-No-strict-graph) to Lnbits Near instant <2 hops Tested up to 150k sats with 100%

Payment G.

Blixt Wallet (V0.6.0-No-strict-graph) to Blixt Wallet (V0.6.1) within 5 seconds <5 hops. Tested up to 100k sats with 100%

Payment H.

Blixt Wallet (V0.6.0-No-strict-graph) to Lnbits.com within 15-25 seconds <4 hops tested up to 150k sats

Notes;

Blixt Wallet (V0.6.1) had 2 time outs while performing the reverse payment from the paying Blixt wallet while all larger amounts above 40k sats failed. also another note is a payment to WoS from Blixt V0.6.1 of 150k sats was also successful. OBW also couldnt pay 100k sats to Blixt Wallet (V0.6.0-No-strict-graph) while connected to same parent node. I did not look further into this and continued with my testing. also china phones are slow... thats about it. lol

5th Test - Forcing MPP through 2 channels - Blixt to Blixt

Sender Blixt v0.6.1 build 30/115 (strict-graph vers), MPP option active
2 channels, with 2 different peers
Android 12
Receiver Blixt v0.6.0 (testing Kevin edition)
1 channel, random node, enough liquidity to send/receive

Proceedings:

before each payment, go to settings and click on "reset payment routing knowledge" and "write config", just to be sure.
keep the Blixt app active and synced, watching the LN peers and node status
saving the channels status after each payment
checking in payment details how many hops and fee was paid

Payment A - test with MPP ON

9 850 sats invoice
available funds: ch.A - 119784, ch.B - 200998
SUCCESS payment, 3 sec, 3 hops, 4 sat fee, only ch.A used
MPP not triggered

Payment B - test with MPP off

9 850 sats invoice
available funds: ch.A - 109929, ch.B - 200998
SUCCESS payment, 2 sec, 3 hops, 4 sat fee, only ch.A used

Payment C - test with MPP ON

120 000 sats invoice
available funds: ch.A - 100075, ch.B - 200998
SUCCESS payment, 7 sec, 4 hops, 68 sat fee, only ch.B used
MPP not triggered

Payment D - test with MPP ON

125 000 sats invoice
available funds: ch.A - 100075, ch.B - 80930
SUCCESS payment, 17 sec, 3 hops, 59 sat fee, both channels used
MPP worked as expected

Payment E - test with MPP ON

tried several amounts between 59 000 and 49 500 - FAILED
40 000 sats invoice
available funds: ch.A - 37552, ch.B - 18393
SUCCESS payment, 12 sec, 4 hops, 19 sat fee, both channels used
MPP worked as expected

Payment F - test with MPP ON to empty the channels

supposedly should be able to send a max 12 582 sats
12 000 sats invoice
available funds: ch.A - 7538, ch.B - 8388
FAILED payment, tried several lower amounts and was impossible to do the payment. Message received: "no router found" or insufficient funds".
seems that LND is keeping a 1% of funds in the channel that CANNOT be moved anymore, due to the fact that we already have "channel reserve", "commit_fee" and "anchor output" already applied over the channels.

Feedback from a Robosater

Android 10, 4GB RAM
Blixt 0.6.1, node ready in 10-20s after start-up
clearnet
wait for graphy sync enabled
MPP enabled

LN Channels

Dunder channel: 500ksats
LNBig-27: 5Msats
Reckless_Apotheosis: 3Msats

Use case

Most of my tx are with Robosats and Boltz nodes. As a rule of thumb, I tend to open channel with a capacity with an order of magnitude higher than my medium size tx. And also I prefer to have at least 2 high capacity channel, as one can be idle it would not bother me. Works like a charm with this config.

6th Test - Circular payments OBW - Blixt A - Blixt B

OBW: 2 HCs + 1 normal ch., v0.1.8-9, clearnet
Blixt A, v0.6.1 (115), 2 ch, clearnet
Blixt B, v0.6.1 (114), 1ch Blixt node, clearnet

Payment A - OBW -> Blixt A (small amount)

9 850 sats
MPP triggered took 9500 sats from 1 ch and 350 sats from a HC
SUCCESS payment done in 5 seconds, 4 sats fee
payment received in 1 ch

Payment B - OBW -> Blixt A (medium amount)

49 900 sats
OBW channels status: HC1 - 55091, HC2 - 11050, NC - 288208
MPP not triggered
SUCCESS payment done in 2 seconds, 19 sats fee
payment received in 1 ch.

Payment C - OBW ->Blixt A (bigger amount)

amounts tried 130000 / 128000 / 125000 / 115000
OBW channels status (can send): HC1 - 55091, HC2 - 11050, NC - 238289
Blixt channels status (can receive): ch.A - 52495, ch.B - 81353 (total 133848)
1st intent (130k) FAILED, reaching 7 parts in flight, timeout after 120 sec
2nd intent (128k) FAILED, reaching 6 parts in flight, timeout after 120 sec
3rd intent, OBW asking about trying again with increased fee, FAILED after 120 sec
4th intent (125k) FAILED, reaching 9 parts in flight, timeout after 120 sec At this moment I force closed both apps to be sure is getting a new fresh graph sync.
5th intent (115k) SUCCESS, after 15 sec ! MPP with 2 parts, 98 sats fee

Payment D - OBW -> Blixt A (left overs)

Restarted both apps, to take a new graph sync

amounts tried 15000 / 13000 / 10000
OBW channels status (can send): HC1 - 55091, HC2 - 1147, NC - 133095
Blixt A channels status (can receive): ch.A - 4895 (ch.res 1244), ch.B - 13953 (ch.res 2100), theoretically a max receive 15000
1st intent (15k) FAILED, 3 parts in flight, timeout after 120 sec
2nd intent (13k) FAILED, 3 parts in flight, timeout after 120 sec
3rd intent (10k) SUCCESS payment done in 15 sec, 1 part, 2 sats fee, received in ch.B Blixt, exited from NC

Payment E - OBW -> Blixt A (pushing to the limits)

Restarted both apps, to take a new graph sync

amounts tried 5000 / 3000
OBW ch. status (can send): HC1 - 45089, HC2 - 1147, NC - 133095
Blixt A ch. status (can receive): ch.A - 4895 (ch.res 1244), ch.B - 3953 (ch.res 2100), so a max total of 5504
1st intent (5k) FAILED , 4 parts in flight, timeout after 120 sec
2nd intent (3k) SUCCESS payment done in 4 sec, 1 part, 3 sats fee, received in ch.A Blixt, exited from HC2

Payment F - OBW -> Blixt A (forcing to fill up ch.B)

amounts tried 1850 / 1000
OBW ch status: HC1 - 45089, HC2 - 29 sat, NC - 131210
Blixt ch. status (can receive: ch.A - 1885 (ch.res 1244), ch.B - 3953 (ch.res 2100)
1st intent 1850 sats FAILED, 3 parts in flight, trying on both channels to receive.
2nd intent 1000 sats SUCCESS, MPP 3 parts, 9 sats fee
Remaining balance (can receive) in Blixt: ch.A - 1385, ch.B - 3463. At this moment I don't know if can receive more. I tried with 500 sats 2 times and the parts are stuck in flight. Nothing appears in Blixt as coming HTLCs.

I personally have many issues when sending from Phoenix (from amount like 12000~15000 sats) to Blixt/OBW/LNBits(my instance, it's working with a Cliché node) even after raising the fee limits, where from OBW or my hosted LNBits/Cliché(HC), and also from custodian services, I often don't have any issue to send from.

hsjoberg / blixt-wallet