Constant high CPU load

[Grix]

I'm running:

• Which Parity version?: Parity//v1.8.6-beta-2d051e4-20180109/x86_64-windows-msvc/rustc1.22.1
• Which operating system?: Windows 10 16299
• How installed?: via installer
• Are you fully synchronized?: yes
• Which network are you connected to?: ethereum
• Did you try to restart the node?: yes

Parity.exe is constantly causing a high CPU load when active, even after syncing is fully done.

The CPU usage hovers between 30 and 50%. Sometimes it spikes higher, but it almost never goes lower. Network and IO usage is also significant. This is with an Intel i5 quad core and SSD.

This has been the case for past versions as well, not just 1.8.6

I can't imagine this is normal behavior, or is running a full node really that resource intensive?

The node log looks pretty much like this:

[1/14/2018, 10:05:43 PM] Imported #4909037 c960…ca62 (60 txs, 8.01 Mgas, 150.31 ms, 16.14 KiB) [1/14/2018, 10:05:37 PM] 16/25 peers 6 MiB chain 118 MiB db 0 bytes queue 167 KiB sync RPC: 1 conn, 15 req/s, 72442 µs [1/14/2018, 10:05:37 PM] Imported #4909036 8e2b…aa98 (235 txs, 8.01 Mgas, 446.04 ms, 36.40 KiB) [1/14/2018, 10:05:33 PM] Imported #4909035 6df4…f4d7 (230 txs, 8.00 Mgas, 266.41 ms, 36.25 KiB) [1/14/2018, 10:05:17 PM] Imported #4909034 787e…320c (233 txs, 7.99 Mgas, 298.78 ms, 32.18 KiB) [1/14/2018, 10:05:07 PM] Imported #4909033 733d…0cee (268 txs, 7.99 Mgas, 501.40 ms, 36.32 KiB) [1/14/2018, 10:04:57 PM] 17/25 peers 6 MiB chain 120 MiB db 0 bytes queue 167 KiB sync RPC: 1 conn, 2 req/s, 367 µs [1/14/2018, 10:04:50 PM] Imported #4909032 61ea…2c33 (262 txs, 8.00 Mgas, 401.61 ms, 34.93 KiB) [1/14/2018, 10:04:36 PM] Imported #4909030 e262…2d55 (202 txs, 5.46 Mgas, 333.79 ms, 26.00 KiB) [1/14/2018, 10:04:32 PM] Imported #4909031 b817…42d8 (165 txs, 8.00 Mgas, 345.86 ms, 30.54 KiB) [1/14/2018, 10:04:24 PM] 15/25 peers 5 MiB chain 120 MiB db 0 bytes queue 167 KiB sync RPC: 1 conn, 2 req/s, 344 µs [1/14/2018, 10:04:04 PM] Imported #4909030 bf50…8038 (186 txs, 7.98 Mgas, 218.27 ms, 30.15 KiB) [1/14/2018, 10:03:53 PM] 13/25 peers 4 MiB chain 120 MiB db 0 bytes queue 167 KiB sync RPC: 1 conn, 0 req/s, 438 µs [1/14/2018, 10:03:51 PM] Imported #4909026 2520…04fc (122 txs, 7.93 Mgas, 282.55 ms, 23.38 KiB) [1/14/2018, 10:03:45 PM] Imported #4909028 8777…9280 (137 txs, 7.95 Mgas, 290.20 ms, 23.56 KiB) [1/14/2018, 10:03:41 PM] Imported #4909027 122f…b902 (131 txs, 7.93 Mgas, 319.74 ms, 22.82 KiB) [1/14/2018, 10:03:37 PM] Reorg to #4909025 3492…ecdc (c202…26af #4909023 09a8…1574 fd85…ccd3) [1/14/2018, 10:03:27 PM] Imported #4909024 df25…9782 (70 txs, 3.38 Mgas, 91.03 ms, 11.50 KiB) [1/14/2018, 10:03:19 PM] 13/25 peers 6 MiB chain 120 MiB db 0 bytes queue 168 KiB sync RPC: 1 conn, 7 req/s, 111 µs [1/14/2018, 10:03:13 PM] Imported #4909024 c202…26af (137 txs, 7.88 Mgas, 326.85 ms, 25.42 KiB) [1/14/2018, 10:03:08 PM] Imported #4909023 09a8…1574 (170 txs, 7.99 Mgas, 432.06 ms, 24.99 KiB) + another 1 block(s) containing 211 tx(s) [1/14/2018, 10:03:04 PM] Reorg to #4909023 09a8…1574 (4b0f…00be #4909021 affb…3fd1 0b3a…990f) [1/14/2018, 10:03:03 PM] Imported #4909022 4b0f…00be (211 txs, 8.00 Mgas, 362.44 ms, 34.66 KiB) [1/14/2018, 10:02:45 PM] 14/25 peers 5 MiB chain 120 MiB db 0 bytes queue 167 KiB sync RPC: 1 conn, 6 req/s, 447 µs [1/14/2018, 10:02:45 PM] Imported #4909018 5277…def4 (206 txs, 7.98 Mgas, 295.08 ms, 35.91 KiB)

[MicahZoltu]

Commenting here rather than a new issue since symptoms are similar, but I can open a new issue if it is determined to be different from this one. The notable information I have is that when looking at Parity in process explorer (Windows 10) I see it has a bunch of threads burning CPU on something called hid_error. This is not normal behavior for my machine.

[MicahZoltu]

Note: My Parity version is Parity//v1.8.2-beta-1b6588c-20171025/x86_64-windows-msvc/rustc1.20.0 on Windows 10.

[MicahZoltu]

Restarting Parity does not resolve the issue. Upon starting up, there are immediately a large number of threads spinning on hid_error.

[MicahZoltu]

Unplugging my phone/mic (both USB devices), restarting Parity, then plugging them back in seems to have resolved the error (though still a lot of threads with that on the stack, perhaps it is a red herring).

[MicahZoltu]

Updating to 1.8.6 resulted in syncing slower than new blocks are coming in.

[MicahZoltu]

Rolling back to 1.7.8 resolves the syncing speed problem, though Parity is still burning a lot of CPU.

[debris]

@Grix @MicahZoltu as a temporary solution, please run parity with --no-hardware-wallets flag. It should help. I will investigate the issue further

[MicahZoltu]

I'm assuming this will mean I will need to restart Parity without the option when I want to use my ledger?

[debris]

yes

[5chdn]

I can't imagine this is normal behavior or is running a full node really that resource intensive?

Yes, your node is processing 15 Ethereum transactions per second.

[Grix]

@5chdn

Yes, your node is processing 15 Ethereum transactions per second.

Doesn't it also do that to all the older transactions when syncing? Yet it does it so much faster. When syncing it burns through around 500 tx/s, while hardly using any more CPU load than the measly 15 tx/s when idle. Doesn't make much sense to me.

And even if it is normal, priority should also be reworked because often the transaction signing box is very slow to appear, sometimes appearing 10+ seconds after I clicked to send a transaction, presumably because of the high CPU load.

Meanwhile bitcoin core averages at like 1% CPU load, each block only causing a tiny spike despite containing thousands of transactions.

[MicahZoltu]

@5chdn I don't think this should be closed without resolution. At least for me, I know the CPU load was abnormally high because I have seen parity on my machine successfully process blocks (during times of congestion) with a stable state usage of ~3-5% utilization. Yesterday I found it using 50-100% CPU and unable to keep up with block processing (was falling behind).

This isn't the first CPU burn problem I have experienced with parity, and the reports come from a variety of users. It really feels like parity should invest some resources in figuring out how to troubleshoot these issues. Perhaps some research on tools that end-users can run when they experience issues like this that will provide the dev team with useful debugging information (e.g. thread profiler).

At this point the only reason I continue to use Parity despite its recurring performance problems is because Geth has worse problems. Problems like this, and your team continuing to not acknowledge them, gives me very little confidence in your product/dev practices.

I'm well aware of how hard it is to debug this class of problems, but there have been enough reports at this point that I feel your team should be devoting effort to actively troubleshooting this issue. So far I have experienced very little engagement from Parity asking me for info despite my continued willingness to provide whatever (within reason) information is needed to troubleshoot or take actions locally to help diagnose the issue.

[5chdn]

Yesterday I found it using 50-100% CPU and unable to keep up with block processing (was falling behind).

After upgrading to 1.8.6+? This is due to the new DB compaction. This will go away with time or if this is annoyance, just reset your DB.

It really feels like parity should invest some resources in figuring out how to troubleshoot these issues.

Unfortunately, we do not have these resources. If anyone is able to diagnose the source of the CPU burn, it will be much appreciated. But currently, the situation for Ethereum client developers is difficult.

Parity is doing an insane task processing the Ethereum chain, state and all the new transactions every second. A CPU is my least concern right now. But as I said, if you figure out there is really an issue, I'm happy to forward it. Speaking of consistently 30-50% CPU usage does not sound like a CPU burn to me.

[MicahZoltu]

No, not after upgrading. Parity has just been running on my computer on version 1.8.2 since it came out, then out of the blue I caught my computer overheating and when I went to look for the source of the problem I saw Parity burning CPU consistently. Prior to this, Parity was using 3-5% with only very brief blips when a new block arrived. Attempting to upgrade to 1.8.6 to see if the problem went away resulted in it actually getting worse (100% rather than 30-50%). Restarting Parity didn't resolve the issue. Rolling back to 1.7.x DID resolve the issue.

This tells me that there is a bug in the 1.8.x line. If I understand you correctly, your theory is that the bug was fixed in 1.8.6 but upon installing 1.8.6 I must wait for compaction to finish to tell since my only symptom is CPU burn and compaction is CPU intensive? If so, I can try installing 1.8.6 again and waiting to see if the problem resolves.

---

Regardless of whether 1.8.6 resolves the issue, I assert with quite a bit of confidence that Parity has historically proven to be quite efficient at block processing and it certainly should not be using massive amounts of CPU steadily during normal operation. I have seen Parity run with a stable CPU utilization of ~5% on my machine for days/weeks on end without significant problem, and during times of congestion (full blocks). When Parity jumps to a stable state CPU utilization of 50%, that tells me something went very wrong internally and this is the class of bug I keep wanting figured out.

[5chdn]

If so, I can try installing 1.8.6 again and waiting to see if the problem resolves.

Yes, please, and plus resetting your DB first.

[MicahZoltu]

I installed Parity 1.8.6 and deleted my %LOCALAPPDATA%\Parity\ folder (which includes all of the chain data/local cache/nodes.json but not my private keys). I then let it sync (warp sync now appears to be fixed) and then "catch up on whatever background work it was doing". It now appears to be running fairly stable with the CPU utilization hovering around 1% when there isn't a new block, and then about 5 seconds of 15% CPU when a new block comes in.

The 15% for 5 second spikes when blocks come in is more than I remember from 1.7.x, but not so high that I'm screaming bloody murder anymore. I will continue to monitor however, because in the past I have also experienced "things look healthy" for a while and then something would happen that would cause everything to go horribly wrong until I wiped my DB again.

[Grix]

It now appears to be running fairly stable with the CPU utilization hovering around 1% when there isn't a new block, and then about 5 seconds of 15% CPU when a new block comes in.

Those number also pretty much confirm that my constant 30+% is not normal, and I'm already on 1.8.6

[MicahZoltu]

@Grix If you are experiencing what I have experienced in the past, then somehow your chain got into a bad state and will stay that way until you wipe out and start over. I have had this happen to me several times in the past with no abnormal usage. Each time I have to wipe out everything and start over, then things are "good for a while". This strategy was really problematic prior to 1.8.6 since Warp Sync was broken, but with that now fixed it isn't as painful as it used to be (about a day for me to get back to synced and stable utilization).

[Grix]

@MicahZoltu Ok I will try that, thanks

[MicahZoltu]

For reference to future readers as to expectations, I have been running Parity 1.8.6 for many days now and things have stabilized such that utilization is under 1% most of the time, with small spikes to 5% (presumably when a block arrives) and an occasional spike to 10%.

The big spikes are 10%, top of the chart is 100%. Window is 10 minutes wide.

@5chdn This is why I (and others) say that we do not expect Parity to be using 30-50% CPU consistently, and even 10% consistently or 15% during new blocks is "unexpected". It isn't because we are bothered by 15%, it is because we know that Parity is capable of staying in sync with much lower numbers so when it is spiking and holding at 50%, we assert that something went wrong internal to Parity (or there is some new attack against the network).

[Grix]

This seems to have been fixed for me in 1.8.10. I now hover around 2% and only reach double digits in spikes when blocks are found, as expected, similar to MicahZoltu's graph.

[kennethhutw]

I also have the same problem. My parity version is version Parity-Ethereum/v2.4.5-stable-76d4064a4-20190408/x86_64-windows-msvc/rustc1.33.0. The CPU usage hovers around 50%