Notify if wallet balance could be incomplete due to utxo limit or backlog history

[gits7r]

The server side part of this problem is logged in electrum-server as 169, but we need to adjust something at the client as well.

When an user tries to query an address with a big backlog history or utxo set bigger than the server limit, the server should return a specific error code that the client will interpret by displaying a warning to the user such as: "One or some of your addresses contain more transactions than the server can process - your balance and transaction history might be incomplete. Use a full node wallet in order to use such addresses." Also this should make the wallet unusable at all for clear reasons, go offline and don't try to connect to that server or other server.

In electrum server we have a limit of 1000 utxo's per address, introduced in commit: d1501bfe0c278304d2bd7d90967a333935044b16 (and we log it after commit: 5e51f038c6e2ab43b22e94912a0f97e083fde321)

I don't know exactly how to handle this for the Sweep function. Is there any way we can spend all utxos of an address for the private key being swept without querying its entire backlog? If we can't sweep it because the utxo set size we should display an error message as well.

[ecdsa]

can you share with me an address that causes this issue?

[gits7r]

@ecdsa Any address that has either too large UTXO set (unspent transactions) either too large spent TXO history. Any popular address would do, for example: 1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp - shouldn't we code the client to interpret a certain error code sent by the server either for the utxo set or spent txo set being overlimit?

[ecdsa]

I imported 1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp in a wallet. len(listunspent()) returns 871, so the limit is not reached. the balance displayed is the same as on blockchain.info

[gits7r]

But does it show the entire history of 3195776 transactions?I am not saying the wallet handles these kind of addresses incorrectly, just that it's not explicit to warn / notify the user by displaying a message if:

• history (spent TXO's list) is not complete due to the configurable server limit (100 or 10000);
• current balance (UTXO set) is not complete due to the server hardcoded limit (1000);

Also, how do we treat the dust UTXO's of an address in both current wallet and sweep function? Might document this to be explicit to the user as well.

[ecdsa]

I agree it would be better to handle these cases explicitly. There's no explicit warning currently, because the protocol was designed before limits were added.

[gits7r]

Right - sorry if I've written this confusing. I guess (low priority) we could adjust 2.7.x to get some error codes from the server (when one of the limits [spent TXOs or UTXOs] is reached, disconnect and warn the user that either the balance either the history backlog is incomplete and should use a full node to properly handle it since it's not possible with our limitations.

Thanks.

[bauerj]

I just tested with the address above.

The wallet just stays on "Synchronizing" forever.

It outputs

[Synchronizer] response error: {'jsonrpc': '2.0', 'id': 73, 'error': {'message': 'response too large (at least 1020631 bytes)', 'code': -32600}, 'method': 'blockchain.scripthash.get_history', 'params': ['1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp']}
[helicarrier.bauerj.eu] connection closed remotely

We should add some kind of error message in this case like proposed by @gits7r

[helpau]

Windows 10, Electrum 4.4.6, 1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp looks empty, no errors or warnings returned. Related bug https://github.com/spesmilo/electrum/issues/2044

[SomberNight]

the most relevant issue is https://github.com/spesmilo/electrum/issues/4315

What server are you testing with @helpau? I cannot reproduce what you are seeing. The expected behaviour is that the client gets stuck syncing and will display it is in "synchronizing" state. If you get to the "green orb" state, that is likely a server issue.

[helpau]

bitcoin.threshold.p2p.org:50002 Fulcrum 1.9.1 server [] tardis.bauerj.eu:50002 ElectrumX 1.10.0 BestEffortRequestFailed eai.coincited.net:50002 ElectrumX 1.16.0 UntrustedServerReturnedError 22jumpstreet.mooo.com:50002 ElectrumX 1.16.0 []

By the way, quite interesting repro steps:

1. Create empty wallet
2. Turn on Select server automatically
3. Add address 1dice8EMZmqKvrGE4Qc9bUFf9PX3xaYDp
4. Go to Console
5. Turn off Select server automatically
6. Select a server which returned error
7. Turn on Select server automatically
8. Look at server rotation and console/logs messages