search

sethforprivacy / self-hosted-services

A core set of privacy-preserving services that can be easily self-hosted via Docker Compose.
192 stars 19 forks source link

Monerod Pinned CPU Usage at 180% (Fully Synced, Full Node) #38

Open reduktr opened 2 years ago

reduktr commented 2 years ago

Hello! I'm running the majority of your stack, but seem to be having an issue with Monerod utilizing too much CPU. For the first 10 hours it runs, it's completely fine and doesn't use more than 4-8% of CPU. (This is after sync, 100%, and running a full node).

After the 10 hours, I get a sharp intake to pretty much a constant 180% CPU usage (4 core, 8GB RAM VPS). I'm not quite sure what's going on? It will stay like this until I restart monerod. Once I restart, it will be okay again for another 10-12 hours, and then it goes into the same pattern. Any help would be appreciated. I can post whatever logs you may require.

Cheers

sethforprivacy commented 2 years ago

@NeonMinnen please post logs from monerod when that happens, would be helpful! If the standard level of logging is insufficient we can increase it as well.

Also please post the exact Docker Compose snippet you're using for monerod if different in any way from the one in the repo.

reduktr commented 2 years ago

@sethforprivacy Hi! Sure thing.

Here is the compose snippet.

monerod:
    image: sethsimmons/simple-monerod:latest
    restart: unless-stopped
    container_name: monerod
    volumes:
      - /mnt/volume-1/bitmonero:/home/monero/.bitmonero
    ports:
      - 18080:18080
      - 18089:18089
    command: >-
      --rpc-restricted-bind-ip=0.0.0.0
      --rpc-restricted-bind-port=18089
      --public-node
      --no-igd
      --no-zmq
      --enable-dns-blocklist
      --limit-rate-up 100000
      --limit-rate-down 100000

Here is also an image of the normal CPU usage to the left, versus when it goes into full CPU hog mode. Taken directly from the monerod CPU usage via Netdata.

Screenshot-from-2022-09-06-22-22-13

Here are some logs that look concerning. I couldn't find anything in between when it actually happened, it seemed to stop logging.

Not sure if anything in here is helpful. I believe this is log_level <1>. I've restarted the container and everything is normal. I'm expecting the same thing to happen again in 10 hours, so I've enabled log_level <2> to get some more logs to post next time it happens.


2022-09-06 04:53:44.913 [P2P0]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 04:53:44.913 [P2P5]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 04:53:44.913 [P2P5]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 04:53:44.913 [P2P5]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 04:53:44.913 [P2P5]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 04:53:44.913 [P2P5]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 04:53:44.913 [P2P5]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 05:29:42.699 [P2P2]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 05:29:42.699 [P2P2]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 05:29:42.699 [P2P2]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 05:29:42.700 [P2P2]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 05:29:42.700 [P2P2]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 05:29:42.700 [P2P2]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 05:43:14.208     7fdd7e6ccb20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:INFO
2022-09-06 05:43:14.208     7fdd7e6ccb20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 05:43:18.525     7f2320d78b20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:INFO
2022-09-06 05:43:18.525     7f2320d78b20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 05:43:19.385     7f2320d78b20        INFO    msgwriter       src/common/scoped_message_writer.h:102  Height: 2705751/2705751 (100.0%) on mainnet, not mining, net hash 2.82 GH/s, v16, 12(out)+58(in) connections, uptime 0d 1h 38m 13s
2022-09-06 05:43:26.662     7f4ef8ca8b20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:INFO
2022-09-06 05:43:26.662     7f4ef8ca8b20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 05:43:27.450     7f4ef8ca8b20        INFO    msgwriter       src/common/scoped_message_writer.h:102  Received 143285549 bytes (136.65 MB) in 81143 packets in 1.6 hours, average 23.71 kB/s = 0.02% of the limit of 97.66 MB/s
2022-09-06 05:43:27.450     7f4ef8ca8b20        INFO    msgwriter       src/common/scoped_message_writer.h:102  Sent 638502725 bytes (608.92 MB) in 58214 packets in 1.6 hours, average 105.67 kB/s = 0.11% of the limit of 97.66 MB/s
2022-09-06 05:44:41.983     7f675ed91b20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:INFO
2022-09-06 05:44:41.983     7f675ed91b20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 05:44:59.104     7f32e65bcb20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:INFO
2022-09-06 05:44:59.104     7f32e65bcb20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 05:44:59.664 [RPC0]  INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:INFO,global:INFO,stacktrace:INFO,logging:INFO,msgwriter:INFO,perf.*:DEBUG
2022-09-06 05:44:59.664 [RPC0]  INFO    perf.daemon.rpc src/common/perf_timer.cpp:156   PERF       71    set_log_level
2022-09-06 05:44:59.664     7f32e65bcb20        INFO    msgwriter       src/common/scoped_message_writer.h:102  Log level is now 1
2022-09-06 05:44:59.666 [P2P6]  INFO    net.p2p.traffic contrib/epee/include/net/levin_protocol_handler_async.h:56      [185.107.95.88:54624 INC] 3779 bytes received for category command-2002 initiated by peer

2022-09-06 06:50:51.654 [P2P3]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 06:50:51.654 [P2P3]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 06:50:51.654 [P2P3]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 07:34:01.893 [P2P1]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 07:34:01.894 [P2P1]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 07:34:01.895 [P2P1]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 07:56:26.889 [P2P4]  INFO    stacktrace      src/common/stack_trace.cpp:133  Exception: boost::wrapexcept<boost::bad_weak_ptr>
2022-09-06 07:56:26.889 [P2P4]  INFO    stacktrace      src/common/stack_trace.cpp:134  Unwound call stack:
2022-09-06 07:56:26.889 [P2P4]  INFO    stacktrace      src/common/stack_trace.cpp:172  
2022-09-06 07:58:20.865     7fd990eb2b20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:I>
2022-09-06 07:58:20.865     7fd990eb2b20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 07:58:22.080     7fd990eb2b20        INFO    msgwriter       src/common/scoped_message_writer.h:102  Received 163183427 bytes (155.62 MB) in 93250 packets in 1.5 hours, average 28.78 kB/s = 0.03% of the limit of 97.66 MB/s
2022-09-06 07:58:22.081     7fd990eb2b20        INFO    msgwriter       src/common/scoped_message_writer.h:102  Sent 1476208529 bytes (1.37 GB) in 87716 packets in 1.5 hours, average 260.36 kB/s = 0.26% of the limit of 97.66 MB/s
2022-09-06 08:28:04.244 [SRV_MAIN]      INFO    global  src/daemon/p2p.h:81     p2p net loop stopped
2022-09-06 08:28:04.245 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:84     Stopping core RPC server...
2022-09-06 08:28:04.245 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:84     Stopping restricted RPC server...
2022-09-06 08:28:04.245 [SRV_MAIN]      INFO    global  src/daemon/daemon.cpp:228       Node stopped.
2022-09-06 08:28:04.245 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:96     Deinitializing core RPC server...
2022-09-06 08:28:04.247 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:96     Deinitializing restricted RPC server...
2022-09-06 08:28:04.249 [SRV_MAIN]      INFO    global  src/daemon/p2p.h:91     Deinitializing p2p...
2022-09-06 08:28:04.268 [SRV_MAIN]      INFO    global  src/daemon/core.h:102   Deinitializing core...
2022-09-06 08:28:04.472 [SRV_MAIN]      INFO    global  src/daemon/protocol.h:75        Stopping cryptonote protocol...
2022-09-06 08:28:04.472 [SRV_MAIN]      INFO    global  src/daemon/protocol.h:79        Cryptonote protocol stopped successfully
2022-09-06 08:28:06.137     7f29c9ca5b20        INFO    logging contrib/epee/src/mlog.cpp:273   New log categories: *:WARNING,net:FATAL,net.http:FATAL,net.ssl:FATAL,net.p2p:FATAL,net.cn:FATAL,daemon.rpc:FATAL,global:INFO,verify:FATAL,serialization:FATAL,daemon.rpc.payment:ERROR,stacktrace:INFO,logging:INFO,msgwriter:I>
2022-09-06 08:28:06.137     7f29c9ca5b20        INFO    global  src/daemon/main.cpp:296 Monero 'Fluorine Fermi' (v0.18.1.0-release)
2022-09-06 08:28:06.138     7f29c9ca5b20        INFO    global  src/daemon/protocol.h:53        Initializing cryptonote protocol...
2022-09-06 08:28:06.138     7f29c9ca5b20        INFO    global  src/daemon/protocol.h:58        Cryptonote protocol initialized OK
2022-09-06 08:28:06.139     7f29c9ca5b20        INFO    global  src/daemon/core.h:64    Initializing core...
2022-09-06 08:28:06.139     7f29c9ca5b20        INFO    global  src/cryptonote_core/cryptonote_core.cpp:519     Loading blockchain from folder /home/monero/.bitmonero/lmdb ...
2022-09-06 08:28:23.637     7f29c9ca5b20        INFO    global  src/cryptonote_core/cryptonote_core.cpp:694     Loading checkpoints
2022-09-06 08:28:23.637     7f29c9ca5b20        INFO    global  src/daemon/core.h:81    Core initialized OK
2022-09-06 08:28:23.637     7f29c9ca5b20        INFO    global  src/daemon/p2p.h:64     Initializing p2p server...
2022-09-06 08:28:23.644     7f29c9ca5b20        INFO    global  src/daemon/p2p.h:69     p2p server initialized OK
2022-09-06 08:28:23.644     7f29c9ca5b20        INFO    global  src/daemon/rpc.h:63     Initializing core RPC server...
2022-09-06 08:28:23.645     7f29c9ca5b20        INFO    global  contrib/epee/include/net/http_server_impl_base.h:79     Binding on 127.0.0.1 (IPv4):18081
2022-09-06 08:28:23.655     7f29c9ca5b20        INFO    global  src/daemon/rpc.h:69     core RPC server initialized OK on port: 18081
2022-09-06 08:28:23.655     7f29c9ca5b20        INFO    global  src/daemon/rpc.h:63     Initializing restricted RPC server...
2022-09-06 08:28:23.656     7f29c9ca5b20        INFO    global  contrib/epee/include/net/http_server_impl_base.h:79     Binding on 0.0.0.0 (IPv4):18089
2022-09-06 08:28:24.013     7f29c9ca5b20        INFO    global  src/daemon/rpc.h:69     restricted RPC server initialized OK on port: 18089
2022-09-06 08:28:24.013     7f29c9ca5b20        INFO    global  src/daemon/rpc.h:74     Starting core RPC server...
2022-09-06 08:28:24.013 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:79     core RPC server started ok
2022-09-06 08:28:24.013 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:74     Starting restricted RPC server...
2022-09-06 08:28:24.013 [SRV_MAIN]      INFO    global  src/daemon/rpc.h:79     restricted RPC server started ok
2022-09-06 08:28:24.013 [SRV_MAIN]      INFO    global  src/daemon/daemon.cpp:214       Public RPC port 18089 will be advertised to other peers over P2P
2022-09-06 08:28:24.013 [SRV_MAIN]      INFO    global  src/daemon/p2p.h:79     Starting p2p net loop...
2022-09-06 08:28:24.333 [P2P9]  INFO    global  src/cryptonote_protocol/cryptonote_protocol_handler.inl:2478    
2022-09-06 08:28:24.333 [P2P9]  INFO    global  src/cryptonote_protocol/cryptonote_protocol_handler.inl:2478    **********************************************************************
2022-09-06 08:28:24.333 [P2P9]  INFO    global  src/cryptonote_protocol/cryptonote_protocol_handler.inl:2478    You are now synchronized with the network. You may now start monero-wallet-cli.
2022-09-06 08:28:24.333 [P2P9]  INFO    global  src/cryptonote_protocol/cryptonote_protocol_handler.inl:2478    
2022-09-06 08:28:24.333 [P2P9]  INFO    global  src/cryptonote_protocol/cryptonote_protocol_handler.inl:2478    Use the "help" command to see the list of available commands.
2022-09-06 08:28:24.333 [P2P9]  INFO    global  src/cryptonote_protocol/cryptonote_protocol_handler.inl:2478    **********************************************************************
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    **********************************************************************
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    The daemon will start synchronizing with the network. This may take a long time to complete.
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    You can set the level of process detailization through "set_log <level|categories>" command,
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    where <level> is between 0 (no details) and 4 (very verbose), or custom category based levels (eg, *:WARNING).
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    Use the "help" command to see the list of available commands.
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    Use "help <command>" to see a command's documentation.
2022-09-06 08:28:25.014 [P2P4]  INFO    global  src/cryptonote_core/cryptonote_core.cpp:1808    
```**********************************************************************
sethforprivacy commented 2 years ago

Thanks for the details!

Only thing that is odd is the stack trace -- do you see that everytime CPU spikes?

The only other thing you could try is removing --public-node from the command args to stop advertising your node publicly and see if that reduces it, but your config of course is normal. If this was a widespread issue I'd be surprised to not have heard from anyone else on this.

reduktr commented 2 years ago

@sethforprivacy

It's strange. When it is doing it's CPU hog thing, it literally is like pinned at 50-55% it doesn't really spike perse.

I really want to run a public node. I don't remember having these issues with the pruned setting, and not running the actual data directory for the blockchain on a hetzner volume. I'm re-syncing a pruned node from scratch, set log level to 2 and going to see if I have the issue after it's done syncing.

Would having the volume on another disk be an issue? I didn't think it would at all. I'm using a volume from Hetzner cloud, as I wanted to run a full node and not take my whole VPS root space up.