[US$ 1500 bounty] Big TX history causing crash & freezes

[kennedyiowa]

With big wallets containing a lot of transactions, Electrum will become unbearably slow, lag, freeze and eventually crash. https://github.com/spesmilo/electrum/issues/2540 is possibly related. My wallet size is 50MB by now, but it started way earlier, probably 10MB.

Version 3.3.8 is the last Electrum version where I can still open my wallet. Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens. The experience below is from 3.3.8 (but earlier versions were affected, too):

Doing some work with the API, def update_wallet(self): is running very very slow as well.

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again. After running Electrum for about 1 day in the background, eventually it will synchronize endlessly (it finishes, then starts to synchronize again immediately), resulting in a crash after doing that for a certain amount of time.

If I had to take a guess, the lags and freezes probably result from Electrum loading and showing all transactions at once. It is possible that having these loaded and drawn as needed could fix the issue.

Electrum-LTC has the same issue.

The only active command I have: wallet.change_gap_limit(1000)

My PC is fast and uses an NVME M.2 SSD.

Since this kind of load would only burden public servers, I'm using my own Electrumx server and node.

While these are bugs and if sufficiently documented, you could probably fix it; the bounty is primarily here because its an edge case and so I don't have to bother with handing you over a wallet that reproduces this. Maybe you can generate a big wallet via Testnet or import a lot of addresses.

The bounty will be valid for 2 months from now. The goal is to fix the freezes and that Electrum 4 cannot even open the wallet. If you think you can tackle this issue and the 2 months are over or need more time, just reply and I can renew it, depending on the situation.

If you are unsure whether this bounty is legitimate or not, just reply below posting status updates to this issue and additionally shortly before its ready to be PR'd. I can make the transaction before your work is delivered. US$ 1500 in XMR or BTC.

[SomberNight]

How many addresses and transactions are there in your wallet? In the Console tab, run these (the results are just examples):

>>> len(wallet.db.transactions)
104
>>> len(wallet.get_addresses())
145

[kennedyiowa]

Electrum BTC:

>> len(wallet.db.transactions)
10000
>> len(wallet.get_addresses())
85000

Electrum LTC:

>> len(wallet.db.transactions)
2000
>> len(wallet.get_addresses())
45000

The BTC one is a few months old wallet. The issues started after a few days or a week already, so the values would've been much lower back then.

[nochiel]

@kennedyiowa: A few questions:

Version 3.3.8 is the last Electrum version where I can still open my wallet. Starting with Electrum 4 (I tried 4.0.3),

Is electrum attempting to update an old wallet? What's the output of get('seed_version') If electrum hangs while trying to upgrade the wallet, then I can write a converter in C.

I haven't been able to synthesise a large enough testnet wallet to reproduce your bug, however, based on a similar experience with an in-house asset editor that made extensive use of json, I am attempting the following (which may be dumb but, without relevant detailed profiling data, it's the only experiment I can think of):

• Get rid of all use of json for the wallet. Instead of storing wallet as compressed, encrypted text, store wallet as a binary blob corresponding to in-memory runtime structs of arrays.
• On startup, convert json wallet to binary blob and write it to disk.

[SomberNight]

Don't try to blindly fix it; it would be a waste of time. You need to be able to reproduce first.

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again

I can confirm this. It's easily reproducible with very large wallets.

Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens

Not sure what changed between 3.3.8 and 4.0.x that could cause this. I can still open my very large wallets fine. @kennedyiowa how are you running Electrum? From source or from official binaries? If binaries, which one? Which OS?

[Transisto]

Would be great to have a very large public wallet that can be used to test? Like a leaked xpub.

On Tue., Nov. 10, 2020, 13:45 ghost43, notifications@github.com wrote:

Don't try to blindly fix it; it would be a waste of time. You need to be able to reproduce first.

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again

I can confirm this. It's easily reproducible with very large wallets.

Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens

Not sure what changed between 3.3.8 and 4.0.x that could cause this. I can still open my very large wallets fine. @kennedyiowa https://github.com/kennedyiowa how are you running Electrum? From source or from official binaries? If binaries, which one? Which OS?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spesmilo/electrum/issues/6625#issuecomment-724892395, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC643CKO5R2FYPI6LPEJHLSPGC33ANCNFSM4SAOIOFQ .

[nochiel]

Electrum runs fine for a minute, freezes completely for 10-20 seconds, then works fine again

I can confirm this. It's easily reproducible with very large wallets.

@SomberNight Does the daemon log indicate when the freeze occurs? Is #6697 the fix for that?

[SomberNight]

The largest wallet I can publicly share is one I have created, for testnet.

Here is the xpub: vpub5VfkVzoT7qgd5gUKjxgGE2oMJU4zKSktusfLx2NaQCTfSeeSY3S723qXKUZZaJzaF6YaF8nwQgbMTWx54Ugkf4NZvSxdzicENHoLJh96EKg

>>> len(wallet.db.transactions)
11012
>>> len(wallet.get_addresses())
10536

Note that multiple things can contribute to freezes at very large wallet size. Many ~trivial operations become resource hogs after a certain threshold. For example, it helps to disable all fiat-related options.

Does the daemon log indicate when the freeze occurs? Is #6697 the fix for that?

That's certainly somewhat related to large wallets, although it has nothing to do with freezes. The freezes are mainly to do with GUI refreshes.

[kennedyiowa]

Apologies for the late reply, I haven't checked this issue daily recently.

@SomberNight Windows 10, downloaded from the official website. Using the installer, since I think this one is faster. To my knowledge, portable is affected, too.

And yes, definitely, the fiat-related options cause a lot of lag as well.

@nochiel Naturally I can only run that command on 3.3.8, the seed version is 18.

In any case, I could just recreate the wallet from seed if the conversion is the issue (correct me if I'm mistaken), but that issue is secondary. In any case, the freezes is the most important one :P

Also, consider this bounty renewed until end of 2020 for now. Of course, like before, if you think you can tackle this issue and the bounty is expired or need more time, just reply and I can renew it, depending on the situation.

[SomberNight]

If you are using the official Windows binaries with such a large wallet, you might be running out of memory. The win binaries we build are 32 bit (x86), they can allocate upto ~4 gigs of memory. (see https://github.com/spesmilo/electrum/issues/6598) The reason 3.3.8 works but 4.0.x does not could be simply due to 4.0.x needing comparatively more memory.

Could you check (using e.g. Task Manager) how much RAM gets allocated by the Electrum process when using 3.3.8?

[kennedyiowa]

Could you check (using e.g. Task Manager) how much RAM gets allocated by the Electrum process when using 3.3.8?

3.3.8 is sitting at 1.5gigs, so we are fine there at least.

[ln2max]

@kennedyiowa please give my PR above a spin, it may fix at least this issue:

it will synchronize endlessly (it finishes, then starts to synchronize again immediately), resulting in a crash after doing that for a certain amount of time.

[kennedyiowa]

@kennedyiowa please give my PR above a spin, it may fix at least this issue:

Thanks a lot for contributing. Sadly the wallet does not even load on Electrum 4.0 for me and I don't really have the time to backport it to 3 and self compile. For other reasons, including security reasons, I also have to wait until this is merged and released officially.

I will certainly honor fixing this side issue to a small degree when I can confirm this was the issue. Alternatively, you could try loading SomberNight's testnet wallet, leave it running for a few days straight (usually took around 1-3 days for me, now with my bigger wallet its within 12 hours), try to get it to crash and then verify your fix is working :)

[ln2max]

On Tue, Nov 17, 2020 at 11:31:19AM -0800, kennedyiowa wrote:

@kennedyiowa please give my PR above a spin, it may fix at least this issue: I will certainly honor fixing this side issue to a small degree when I can confirm this was the issue. Alternatively, you could try loading SomberNight's testnet wallet, leave it running for a few days straight, try to get it to crash and then verify your fix is working :)

SomberNight's testnet wallet loads and syncs up just fine for me using the above PR (electrum 4.0.4) in a Linux VM using public servers. I have 4GB RAM allocated to the VM but it is only using about 1.3 GB.

I don't have a private testnet server available for testing other scenarios.

It does continually have to re-connect and re-sync, but (thanks to the randomization logic added in the PR?) all the transactions in the wallet load fairly early on and it should be quite usable.

I will leave it running for a while and see if it crashes.

[Transisto]

Would it make sense to have a parameter ex. 'monitorlast 10000' that only check last 10000 used addresses and the gap limit.

On Thu., Nov. 19, 2020, 08:15 ln2max, notifications@github.com wrote:

On Tue, Nov 17, 2020 at 11:31:19AM -0800, kennedyiowa wrote:

@kennedyiowa please give my PR above a spin, it may fix at least this issue: I will certainly honor fixing this side issue to a small degree when I can confirm this was the issue. Alternatively, you could try loading SomberNight's testnet wallet, leave it running for a few days straight, try to get it to crash and then verify your fix is working :)

SomberNight's testnet wallet loads and syncs up just fine for me using the above PR (electrum 4.0.4) in a Linux VM using public servers. I have 4GB RAM allocated to the VM but it is only using about 1.3 GB.

I don't have a private testnet server available for testing other scenarios.

It does continually have to re-connect and re-sync, but (thanks to the randomization logic added in the PR?) all the transactions in the wallet load fairly early on and it should be quite usable.

I will leave it running for a while and see if it crashes.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spesmilo/electrum/issues/6625#issuecomment-730366957, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC643EZTDRI5YY7MISJ7FTSQUK6DANCNFSM4SAOIOFQ .

[ln2max]

On Thu, Nov 19, 2020 at 05:28:30AM -0800, Transisto wrote:

Would it make sense to have a parameter ex. 'monitorlast 10000' that only check last 10000 used addresses and the gap limit.

That has been proposed elsewhere in other issues (see the discussions referenced in my PR).

[ln2max]

On Tue, Nov 17, 2020 at 11:31:19AM -0800, kennedyiowa wrote:

I will certainly honor fixing this side issue to a small degree when I can confirm this was the issue. Alternatively, you could try loading SomberNight's testnet wallet, leave it running for a few days straight, try to get it to crash and then verify your fix is working :)

So far the wallet has not crashed. A bit before 21:14 UTC (running since my last post) the sync stopped at 56/10536 addresses. The UI was still responsive, however. I'm re-running the test with electrum in verbose mode to see what the actual issue is.

[ln2max]

Ok, it looks like after many many hours of running, electrum stops refreshing balances at between 50 and 80 addresses into the 10000 or so of the full @SomberNight wallet.

I've got the full verbose console output logged to the attached verbose.log file, the last sync attempt ends at about line 24884 in the file.

I don't see anything in the log which would indicate what the problem is.

Overall the user experience is quite alright, even after the sync stops, Electrum is still responsive and restarting it fixes the issue.

@kennedyiowa since the UX with SomberNight's testnet wallet is fine, it may be worth exporting the xpub (for testing) from your wallet and importing it into Electrum 4 that way -- rather than the wallet itself, Electrum may be choking on importing your wallet. You may find that the only real bug is having to restart the wallet every day, which is an annoyance but not a showstopper.

[kennedyiowa]

@kennedyiowa since the UX with SomberNight's testnet wallet is fine, it may be worth exporting the xpub (for testing) from your wallet and importing it into Electrum 4 that way -- rather than the wallet itself, Electrum may be choking on importing your wallet. You may find that the only real bug is having to restart the wallet every day, which is an annoyance but not a showstopper.

Good idea. Just did that on 4.0.5, and apparently the issue wasn't the wallet. 4.x doesn't open at all even in a clean state. Wiped the Electrum appdata folder, started Electrum and it just hangs in the background forever like I mentioned in the original issue. Even reinstalled Windows since I opened this issue. Pretty weird since I don't see any other issues like this on here.

Restarting Electrum every day one or two times is only a small annoyance compared to the freezes. I work with the wallet a lot and need to check transactions. Usually within 60 seconds it will become fully unresponsive for about 10-20 seconds. Then works for another 60 seconds until it repeats. This is costing me more brain cells than the bounty is worth.

@ln2max This freeze issue is not occuring at all on your system? Are you using Windows? Maybe version 4 did fix the freezes, but I wouldn't assume so, otherwise SomberNight wouldn't have confirmed it. Or your changes fixed it, but I really don't see that this was the cause of the freezes.

[ln2max]

On Sat, Nov 21, 2020 at 08:18:50PM -0800, kennedyiowa wrote:

Good idea. Just did that on 4.0.5, and apparently the issue wasn't the wallet. 4.x doesn't open at all even in a clean state. Wiped the Electrum appdata folder, started Electrum and it just hangs in the background forever like I mentioned in the original issue. Even reinstalled Windows since I opened this issue. Pretty weird since I don't see any other issue like this.

Sounds like you found the problem :)

Restarting Electrum every day one or two times is only a small annoyance compared to the freezes. I work with the wallet a lot and need to check transactions. Usually within 60 seconds it will become fully unresponsive for about 10-20 seconds. Then works for another 60 seconds until it repeats. This is costing me more brain cells than the bounty is worth.

Well, using the SomberNight wallet on Linux I experienced nothing like that "works for 60 seconds, unresponsive for 10-20 seconds" behavior after my fixes. I did notice that behavior in earlier testing, but that could have been on Electrum 3.8 -- I didn't keep notes.

@ln2max This freeze issue is not occuring at all on your system? Are you using Windows? Maybe version 4 did fix the freezes, but I wouldn't assume so, otherwise SomberNight wouldn't have confirmed it. Or your changes fixed it, but I really don't see that this was the cause of the freezes.

I'm using a Linux (Debian 10) virtual machine for my tests, not Windows. I use a virtual machine for security reasons (it provides isolation when testing unreviewed and untrusted code). I don't have a Windows machine to test this with.

You might consider doing the same thing on your end -- installing Linux as a virtual machine (e.g using Virtualbox) and try running Electrum on that.

You can get Virtualbox here: https://www.virtualbox.org/

There are also pre-packaged ready-to-go Linux images for Virtualbox available... these should not be trusted for serious work, but for testing they should do the job. (I have not tested any of these images, my VMs are built in a different way)

Debian 10 is the flavor of Linux I have been using: https://www.linuxvmimages.com/images/debian-10/

If you are unfamiliar with Linux then you may find Ubuntu or Linux Mint easier to get started with: https://www.linuxvmimages.com/images/ubuntu-2004/ https://www.linuxvmimages.com/images/linuxmint-20/

If switching to Linux fixes it, it's probably worth buying a cheap Lenovo Thinkpad off ebay (or, depending on your threat model, finding a new laptop that's known to support Linux well), installing Linux on it, and using that as your "Electrum machine"..

[SomberNight]

Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens. Just did that on 4.0.5, and apparently the issue wasn't the wallet. 4.x doesn't open at all even in a clean state

Uhmm, that's weird.

So you are on Windows... could you please try using the setup/installer exe? It should unpack two executables in the install folder: one should be named e.g. electrum-4.0.5-debug.exe. Try starting that, from cmd/powershell, with -v, so e.g.:

PS C:\Program Files (x86)\Electrum> .\electrum-4.0.5-debug.exe -v

Anything interesting in the logs there?

[sureplus1988]

Hello Spesmilo/Electrum my name is antonio Louis Perez and I am sorry to inform you that you have not been talking to me as the owner and founder of this email and or currency that you are speaking of....my account has been compromised and I would like to freeze all of my currency and inclosed assets until I am reached at my cell phone number +1(713)315-0659... some how or another a person has been using my email thru another phone I believe it's called cloning... and if there is anyway shape or form to prosecute and convict these prisoners this in individual i am willing to take this matter as far and as long as it takes until this matter is resolved i believe it is either my My ex girlfriend adriana troutman My current girlfriend deidra Coker Or her son in law Matthew caulburth Thank you for you time to read and review my reply.... Antonio Louis Perez

On Wed, Nov 25, 2020, 2:18 AM ghost43 notifications@github.com wrote:

Starting with Electrum 4 (I tried 4.0.3), Electrum won't even open anymore, the process is running but nothing happens. Just did that on 4.0.5, and apparently the issue wasn't the wallet. 4.x doesn't open at all even in a clean state

Uhmm, that's weird.

So you are on Windows... could you please try using the setup/installer exe? It should unpack two executables in the install folder: one should be named e.g. electrum-4.0.5-debug.exe. Try starting that, from cmd/powershell, with -v, so e.g.:

PS C:\Program Files (x86)\Electrum> .\electrum-4.0.5-debug.exe -v

Anything interesting in the logs there?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/spesmilo/electrum/issues/6625#issuecomment-733543608, or unsubscribe https://github.com/notifications/unsubscribe-auth/APEBMGK7JTHYUHZAHTJHUZ3SRS4UVANCNFSM4SAOIOFQ .

[kennedyiowa]

Anything interesting in the logs there?

Late again. Its just stuck here (wiped AppData folder, it recreated it):

I | logging | Electrum version: 4.0.5 - https://electrum.org - https://github.com/spesmilo/electrum
I | logging | Python version: 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:01:55) [MSC v.1900 32 bit (Intel)]. On platform: Windows-10-10.0.19041-SP0
I | logging | Logging to file: None
I | logging | Log filters: verbosity '*', verbosity_shortcuts ''
I/p | plugin.Plugins | registering hardware bitbox02: ('hardware', 'bitbox02', 'BitBox02')
I/p | plugin.Plugins | registering hardware coldcard: ('hardware', 'coldcard', 'Coldcard Wallet')
I/p | plugin.Plugins | registering hardware digitalbitbox: ('hardware', 'digitalbitbox', 'Digital Bitbox wallet')
I/p | plugin.Plugins | registering hardware keepkey: ('hardware', 'keepkey', 'KeepKey wallet')
I/p | plugin.Plugins | registering hardware ledger: ('hardware', 'ledger', 'Ledger wallet')
I/p | plugin.Plugins | registering hardware safe_t: ('hardware', 'safe_t', 'Safe-T mini wallet')
I/p | plugin.Plugins | registering hardware trezor: ('hardware', 'trezor', 'Trezor wallet')
I/p | plugin.Plugins | registering wallet type ('2fa', 'trustedcoin')
D | util.profiler | Plugins.__init__ 0.0579
I/n | network | blockchains [0]
I | exchange_rate.FxThread | using exchange CoinGecko
I/n | network | starting network
I/n | network | setting proxy None
I/n | network | connecting to blockstream.info:700:s as new interface
I/n | network | starting taskgroup.
I | channel_db.ChannelDB | SQL thread started
D | util.profiler | Daemon.__init__ 1.1162
I | daemon.Daemon | launching GUI: qt
I | channel_db.ChannelDB | Creating database
I | daemon.Daemon | starting taskgroup.
I | gui.qt.history_list | could not import electrum.plot. This feature needs matplotlib to be installed.
I | channel_db.ChannelDB | load data 0 0 0
I | channel_db.ChannelDB | num_channels_partitioned_by_policy_count. 0p: 0, 1p: 0, 2p: 0
D | util.profiler | ChannelDB.load_data 0.0020

[SomberNight]

@kennedyiowa looks like it does not even get to the very early steps of initialising the GUI... https://github.com/spesmilo/electrum/blob/78513affe536ba7d8150dc2ee044e3258097317f/electrum/daemon.py#L569 https://github.com/spesmilo/electrum/blob/78513affe536ba7d8150dc2ee044e3258097317f/electrum/gui/qt/__init__.py#L89-L95

but that's not all... the event loop thread (e.g. network) is not making progress either.

[SomberNight]

Could you please try running with --offline? so e.g.:

PS C:\Program Files (x86)\Electrum> .\electrum-4.0.5-debug.exe -v --offline

[kennedyiowa]

It changes to this:

I | logging | Electrum version: 4.0.5 - https://electrum.org - https://github.com/spesmilo/electrum
I | logging | Python version: 3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 18:01:55) [MSC v.1900 32 bit (Intel)]. On platform: Windows-10-10.0.19041-SP0
I | logging | Logging to file: None
I | logging | Log filters: verbosity '*', verbosity_shortcuts ''
I/p | plugin.Plugins | registering hardware bitbox02: ('hardware', 'bitbox02', 'BitBox02')
I/p | plugin.Plugins | registering hardware coldcard: ('hardware', 'coldcard', 'Coldcard Wallet')
I/p | plugin.Plugins | registering hardware digitalbitbox: ('hardware', 'digitalbitbox', 'Digital Bitbox wallet')
I/p | plugin.Plugins | registering hardware keepkey: ('hardware', 'keepkey', 'KeepKey wallet')
I/p | plugin.Plugins | registering hardware ledger: ('hardware', 'ledger', 'Ledger wallet')
I/p | plugin.Plugins | registering hardware safe_t: ('hardware', 'safe_t', 'Safe-T mini wallet')
I/p | plugin.Plugins | registering hardware trezor: ('hardware', 'trezor', 'Trezor wallet')
I/p | plugin.Plugins | registering wallet type ('2fa', 'trustedcoin')
D | util.profiler | Plugins.__init__ 0.0589
I | exchange_rate.FxThread | using exchange CoinGecko
D | util.profiler | Daemon.__init__ 0.0040
I | daemon.Daemon | launching GUI: qt
I | daemon.Daemon | starting taskgroup.
I | gui.qt.history_list | could not import electrum.plot. This feature needs matplotlib to be installed.

[amuna1225]

Hi all, i'm having this issue on windows 10 with electrum ltc, have had the issue through numerous updates and am currently on 4.0.9.3 and the issue is still present. What an earlier poster said regarding how it freezes for 10-20 seconds then comes back to life is accurate..but eventually it doesnt come back to life at all and i'm forced to restart the software..i have tried using every version (portable, installer, exe, ect) and every single one freezes. My wallet has about 6000 transactions in it. I have noticed and haven't seen this mentioned that 90% of the time this happens is when a transaction is being sent/received or the server is trying to update my wallet to the next block. I am seriously considering using another wallet for LTC at this point with how often this freezes for me and how annoying and time consuming it is. I run Dash electrum also and have never had any issues, especially not an issue like this..maybe compare the code between dash and ltc electrum and you might spot the problem?

[SomberNight]

@amuna1225 Presumably the reason it does not happen for you with the dash fork is that your wallet there does not have a long history.

There are multiple causes of freezes with large wallets but the primary problem is that upon almost every update the whole list is recreated and redrawn. It is not just the History tab, but also the Addresses and Coins tabs. (Although only the currently visible tab is refreshed on updates, the others are only refreshed on demand when switching to them)

EDIT: To be clear, a viable workaround is to simply create a new wallet/seed and send all your coins there.