Out of memory while synchronizing the node

[SixoneKui]

I tried run a node on a mounted volume using: docker run -it -v /mnt/HC_Volume_37317682:/acala/data acala/acala-node:2.21.0 --base-path=/acala/data --chain=acala

the above is got from another question. But I see acala-node requires a large RAM, sometimes over 32GB and then causes out of memory issue (My could server has only 32GB ). Is this normal ?

Thanks a lot.

[xlc]

Do you have logs, memory usage matrices etc from the server?

[SixoneKui]

Please see this. It has used 49GB and this number will increase till out of memory (64GB).

[xlc]

This is not enough for us to diagnose the issue. I need logs and memory usage growth matrix. Also what’s the Acala node version, OS version?

[SixoneKui]

ubuntu 22.04.3 LTS acala node version 2.21.0 (should I use 2.20.0?)

Logs:

My RAM is 64GB Ram PID USER PR NI VIRT RES SHR S CPU MEM TIME+ COMMAND 20605 1000 20 0 252.2g 38.5g 47512 D 43.8 62.7 6:43.89 acala 20605 1000 20 0 252.2g 38.8g 47512 D 33.9 63.3 6:47.28 acala 20605 1000 20 0 252.2g 39.1g 47512 R 34.2 63.8 6:50.71 acala 20605 1000 20 0 252.2g 39.5g 47512 R 32.3 64.4 6:53.95 acala 20605 1000 20 0 252.2g 39.8g 47512 D 33.4 64.9 6:57.30 acala 20605 1000 20 0 252.2g 40.1g 47512 D 32.4 65.5 7:00.54 acala 20605 1000 20 0 252.2g 40.5g 47512 D 31.4 66.0 7:03.69 acala 20605 1000 20 0 258.2g 40.8g 47512 R 32.1 66.5 7:06.91 acala 20605 1000 20 0 258.2g 41.1g 47512 R 32.5 67.1 7:10.17 acala 20605 1000 20 0 258.2g 41.5g 47512 D 32.5 67.6 7:13.43 acala 20605 1000 20 0 258.2g 41.8g 47512 D 33.1 68.2 7:16.75 acala 20605 1000 20 0 258.2g 42.1g 47512 D 33.1 68.7 7:20.06 acala 20605 1000 20 0 258.2g 42.5g 47512 D 33.2 69.3 7:23.39 acala 20605 1000 20 0 258.2g 42.8g 47512 D 31.7 69.8 7:26.57 acala 20605 1000 20 0 258.2g 43.1g 47512 D 33.9 70.4 7:29.97 acala 20605 1000 20 0 258.2g 43.5g 47512 R 33.3 70.9 7:33.31 acala 20605 1000 20 0 258.2g 43.8g 47512 D 31.7 71.4 7:36.48 acala 20605 1000 20 0 258.2g 44.1g 47512 D 33.4 72.0 7:39.83 acala 20605 1000 20 0 258.2g 44.5g 47496 R 32.4 72.5 7:43.08 acala 20605 1000 20 0 258.2g 44.8g 47496 R 32.3 73.0 7:46.31 acala 20605 1000 20 0 258.3g 45.1g 47496 D 32.8 73.6 7:49.60 acala 20605 1000 20 0 258.3g 45.5g 47496 R 31.3 74.1 7:52.74 acala 20605 1000 20 0 258.3g 45.8g 47496 R 32.6 74.7 7:56.01 acala 20605 1000 20 0 258.3g 46.1g 47496 D 33.5 75.2 7:59.37 acala 20605 1000 20 0 258.3g 46.4g 47496 R 32.3 75.7 8:02.60 acala 20605 1000 20 0 265.3g 46.8g 47496 D 34.0 76.3 8:06.01 acala 20605 1000 20 0 265.3g 47.1g 47496 D 33.3 76.8 8:09.35 acala 20605 1000 20 0 265.3g 47.4g 47496 R 32.8 77.3 8:12.64 acala 20605 1000 20 0 265.3g 47.7g 47496 R 32.6 77.9 8:15.91 acala 20605 1000 20 0 265.3g 48.1g 47496 R 32.7 78.4 8:19.18 acala 20605 1000 20 0 265.3g 48.4g 47496 D 32.8 78.9 8:22.47 acala 20605 1000 20 0 265.3g 48.7g 47496 R 33.1 79.5 8:25.79 acala 20605 1000 20 0 265.3g 49.0g 47496 R 32.8 80.0 8:29.08 acala 20605 1000 20 0 265.3g 49.4g 47496 D 34.0 80.5 8:32.48 acala 20605 1000 20 0 265.3g 49.7g 47496 R 31.9 81.1 8:35.68 acala 20605 1000 20 0 265.3g 50.0g 47496 D 33.9 81.6 8:39.07 acala 20605 1000 20 0 265.3g 50.4g 47496 D 32.9 82.1 8:42.37 acala 20605 1000 20 0 265.3g 50.7g 47496 D 33.8 82.7 8:45.76 acala 20605 1000 20 0 265.3g 51.0g 47496 D 34.2 83.3 8:49.19 acala 20605 1000 20 0 265.3g 51.4g 47496 D 34.4 83.8 8:52.64 acala 20605 1000 20 0 265.3g 51.7g 47496 D 35.9 84.4 8:56.23 acala 20605 1000 20 0 265.3g 52.1g 47496 D 35.4 84.9 8:59.78 acala 20605 1000 20 0 265.3g 52.4g 47496 D 33.8 85.5 9:03.17 acala 20605 1000 20 0 265.3g 52.7g 24672 D 36.1 86.0 9:06.79 acala 20605 1000 20 0 654.2g 53.5g 54700 S 208.0 87.3 9:27.61 acala 20605 1000 20 0 654.6g 53.6g 54740 S 236.1 87.5 9:51.27 acala 20605 1000 20 0 654.8g 53.5g 54676 S 256.0 87.2 10:16.92 acala 20605 1000 20 0 654.9g 53.2g 54712 S 262.3 86.8 10:43.20 acala 20605 1000 20 0 655.0g 53.0g 54712 S 257.3 86.5 11:08.98 acala 20605 1000 20 0 655.1g 52.8g 54712 S 247.5 86.0 11:33.75 acala 20605 1000 20 0 655.2g 52.1g 54712 S 229.5 85.0 11:56.75 acala 20605 1000 20 0 655.2g 51.7g 54716 S 106.5 84.3 12:07.42 acala 20605 1000 20 0 655.2g 51.3g 54716 S 104.3 83.6 12:17.87 acala 20605 1000 20 0 655.2g 50.9g 54716 S 102.8 83.0 12:28.17 acala 20605 1000 20 0 655.2g 50.5g 54716 S 102.5 82.4 12:38.43 acala 20605 1000 20 0 655.2g 50.2g 54716 S 102.6 81.8 12:48.71 acala 20605 1000 20 0 655.2g 49.8g 54716 S 102.2 81.2 12:58.95 acala 20605 1000 20 0 669.2g 49.6g 54716 S 102.4 80.8 13:09.20 acala 20605 1000 20 0 669.2g 49.2g 54780 S 102.9 80.3 13:19.51 acala 20605 1000 20 0 669.2g 48.9g 54780 S 102.1 79.7 13:29.73 acala 20605 1000 20 0 669.2g 48.5g 54844 S 102.5 79.2 13:40.00 acala 20605 1000 20 0 669.2g 49.2g 55012 S 112.6 80.2 13:51.28 acala 20605 1000 20 0 685.2g 53.6g 55012 S 117.0 87.5 14:03.00 acala

[xlc]

Your screenshot is unreadable. Please just copy & paste the logs instead of screenshots. We don't have any known issue on 2.21. But you could try downgrade 2.20/2.19 to see if it helps.

[SixoneKui]

I have switched to 2.20. This issue still occurs . See the log below:

2023-10-05 02:29:42 [Parachain] ✨ Imported #4600692 (0xc7d6…cd25) 2023-10-05 02:39:28 Acala Node 2023-10-05 02:39:28 ✌️ version 2.20.0-unknown 2023-10-05 02:39:28 ❤️ by Acala Developers, 2019-2023 2023-10-05 02:39:28 📋 Chain specification: Acala 2023-10-05 02:39:28 🏷 Node name: kkc 2023-10-05 02:39:28 👤 Role: FULL 2023-10-05 02:39:28 💾 Database: RocksDb at /acala/data/chains/acala/db/full 2023-10-05 02:39:28 Parachain id: Id(2000) 2023-10-05 02:39:28 Parachain id: Id(2000) 2023-10-05 02:39:28 Is collating: no 2023-10-05 02:45:55 [Relaychain] 🏷 Local node identity is: 12D3KooWFu3t8kw9PEefyAcz1oAndfits5nsLmHkeNxk5QP2JjYB 2023-10-05 02:45:56 [Relaychain] 💻 Operating system: linux 2023-10-05 02:45:56 [Relaychain] 💻 CPU architecture: x86_64 2023-10-05 02:45:56 [Relaychain] 💻 Target environment: gnu 2023-10-05 02:45:56 [Relaychain] 💻 CPU: Intel Xeon Processor (Skylake, IBRS) 2023-10-05 02:45:56 [Relaychain] 💻 CPU cores: 4 2023-10-05 02:45:56 [Relaychain] 💻 Memory: 15611MB 2023-10-05 02:45:56 [Relaychain] 💻 Kernel: 5.15.0-84-generic 2023-10-05 02:45:56 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.4 LTS 2023-10-05 02:45:56 [Relaychain] 💻 Virtual machine: yes 2023-10-05 02:45:56 [Relaychain] 📦 Highest known block at #8274944 2023-10-05 02:45:56 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616 2023-10-05 02:45:56 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost:", "http://127.0.0.1:", "https://localhost:", "https://127.0.0.1:", "https://polkadot.js.org"] 2023-10-05 02:45:56 [Relaychain] Starting with an empty approval vote DB. 2023-10-05 02:45:56 [Parachain] 🏷 Local node identity is: 12D3KooWRmbcLPAirR1eVvvUxg7xaeGRiiPY2FRYSjB2qevmRRFR 2023-10-05 02:45:56 [Parachain] 💻 Operating system: linux 2023-10-05 02:45:56 [Parachain] 💻 CPU architecture: x86_64 2023-10-05 02:45:56 [Parachain] 💻 Target environment: gnu 2023-10-05 02:45:56 [Parachain] 💻 CPU: Intel Xeon Processor (Skylake, IBRS) 2023-10-05 02:45:56 [Parachain] 💻 CPU cores: 4 2023-10-05 02:45:56 [Parachain] 💻 Memory: 15611MB 2023-10-05 02:45:56 [Parachain] 💻 Kernel: 5.15.0-84-generic 2023-10-05 02:45:56 [Parachain] 💻 Linux distribution: Ubuntu 20.04.4 LTS 2023-10-05 02:45:56 [Parachain] 💻 Virtual machine: yes 2023-10-05 02:45:56 [Parachain] 📦 Highest known block at #4600688 2023-10-05 02:45:56 [Parachain] Running JSON-RPC server: addr=0.0.0.0:9933, allowed origins=["*"] 2023-10-05 02:45:56 [Parachain] 〽️ Prometheus exporter started at 127.0.0.1:9615 2023-10-05 02:45:56 [Relaychain] di Prometheus exporter started at 127.0.0.1:9615 2023-10-05 02:45:56 [Parachain] discovered: 12D3KooWFu3t8kw9PEefyAcz1oAndfits5nsLmHkeNxk5QP2JjYB /ip4/172.17.0.2/tcp/30334/ws 2023-10-05 02:45:56 [Parachain] ✨ Imported #4600693 (0x69e4…d78c) 2023-10-05 02:45:56 [Relaychain] 🔍 Discovered new external address for our node: /ip4/95.216.220.103/tcp/30334/ws/p2p/12D3KooWFu3t8kw9PEefyAcz1oAndfits5nsLmHkeNxk5QP2JjYB 2023-10-05 02:45:56 [Parachain] 🔍 Discovered new external address for our node: /ip4/95.216.220.103/tcp/30333/ws/p2p/12D3KooWRmbcLPAirR1eVvvUxg7xaeGRiiPY2FRYSjB2qevmRRFR 2023-10-05 02:46:00 [Parachain] ✨ Imported #4600770 (0x9220…6988) 2023-10-05 02:46:01 [Relaychain] ⚙️ Syncing, target=#17583135 (8 peers), best: #8275064 (0x4d4b…971f), finalized #8274432 (0x651c…8f51), ⬇ 7.7MiB/s ⬆ 67.2kiB/s 2023-10-05 02:46:01 [Parachain] 💤 Idle (8 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 244.9kiB/s ⬆ 8.1kiB/s 2023-10-05 02:46:02 [Parachain] ✨ Imported #4600771 (0xfe34…77c4) 2023-10-05 02:46:06 [Relaychain] ⚙️ Syncing 31.6 bps, target=#17583135 (9 peers), best: #8275222 (0xa1fd…00c1), finalized #8274432 (0x651c…8f51), ⬇ 778.3kiB/s ⬆ 50.2kiB/s 2023-10-05 02:46:06 [Parachain] 💤 Idle (8 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 11.8kiB/s ⬆ 3.4kiB/s 2023-10-05 02:46:11 [Relaychain] ⚙️ Syncing 32.2 bps, target=#17583135 (11 peers), best: #8275383 (0x9e2a…c5d1), finalized #8274432 (0x651c…8f51), ⬇ 182.1kiB/s ⬆ 82.6kiB/s 2023-10-05 02:46:11 [Parachain] 💤 Idle (9 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 10.4kiB/s ⬆ 2.5kiB/s 2023-10-05 02:46:13 [Parachain] ✨ Imported #4600772 (0x62fa…29c6) 2023-10-05 02:46:16 [Relaychain] ⚙️ Syncing 14.5 bps, target=#17583135 (13 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 89.7kiB/s ⬆ 52.5kiB/s 2023-10-05 02:46:16 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 12.2kiB/s ⬆ 12.0kiB/s 2023-10-05 02:46:21 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583135 (17 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 129.5kiB/s ⬆ 53.8kiB/s 2023-10-05 02:46:21 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0 ⬆ 0 202 3-10-05 02:46:26 [Parachain] ✨ Imported #4600773 (0x5c59…127f) 2023-10-05 02:46:26 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583136 (21 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 94.7kiB/s ⬆ 50.3kiB/s 2023-10-05 02:46:26 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.5kiB/s ⬆ 0.8kiB/s 2023-10-05 02:46:26 [Parachain] ✨ Imported #4600773 (0xa24f…1c07) 2023-10-05 02:46:31 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583137 (25 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 165.0kiB/s ⬆ 66.2 2kiB/s 2023-10-05 02:46:31 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.8kiB/s ⬆ 6.4kiB/s 2023-10-05 02:46:36 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583138 (29 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 141.6kiB/s ⬆ 59.3kiB B/s 2023-10-05 02:46:36 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 69 B/s ⬆ 5.4kiB/s 2023-10-05 02:46:36 [Parachain] ✨ Imported #4600774 (0xe280…6094) 2023-10-05 02:446:41 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583139 (37 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 117.2kiB/s ⬆ 49.5kiB/s 2023-10-05 02:46:41 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 14.6kiB/s ⬆ 11.6kiB/s 2023-10-05 02:46:46 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583139 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 94.2kiB/s ⬆ 42.2kiB/s 2023-10-05 02:46:46 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.3kiB/s ⬆ 0.3kiB/s 2023-10-05 02:46:48 [Parachain] ✨ Imported #4600775 (0x6fc1…6340) 2023-10-05 02:46:48 [Parachain] ✨ Imported #4600775 (0xd526…a2d5) 2023-10-05 02:46:51 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583141 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 124.2kiB/s ⬆ 69.8kiB/s 2023-10-05 02:46:51 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 20.8kiB/s ⬆ 15.3kiB/s 2023-10-05 02:46:56 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 155.8kiB/s ⬆ 119.5kiB/s 2023-10-05 02:46:56 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.1kiB/s ⬆ 0.1kiB/s 2023-10-05 02:47:01 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 331.9kiB/s ⬆ 209.6kiB/s 2023-10-05 02:47:01 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 7.5kiB/s ⬆ 2.6kiB/s 2023-10-05 02:47:02 [Parachain] ✨ Imported #4600776 (0x9c67…c1b3) 2023-10-05 02:47:06 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 225.0kiB/s ⬆ 210.5kiB/s 2023-10-05 02:47:06 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.3kiB/s ⬆ 6.2kiB/s 2023-10-05 02:47:11 [Relaychain] ⚙️ Syncing 0.0 bps, target=#17583142 (40 peers), best: #8275456 (0xf374…335f), finalized #8274432 (0x651c…8f51), ⬇ 453.6kiB/s ⬆ 331.9kiB/s 2023-10-05 02:47:11 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 12 B/s ⬆ 31 B/s 2023-10-05 02:47:12 [Parachain] ✨ Imported #4600777 (0x9bcc…b8f3) 2023-10-05 02:47:16 [Relaychain] ⚙️ Syncing 3.7 bps, target=#17583142 (40 peers), best: #8275475 (0xc218…3723), finalized #8275456 (0xf374…335f), ⬇ 1.2MiB/s ⬆ 285.1kiB/s 2023-10-05 02:47:16 [Parachain] 💤 Idle (10 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 31.6kiB/s ⬆ 17.9kiB/s 2023-10-05 02:47:21 [Relaychain] ⚙️ Syncing 30.0 bps, target=#17583143 (40 peers), best: #8275625 (0x9db1…21de), finalized #8275456 (0xf374…335f), ⬇ 910.9kiB/s ⬆ 346.3kiB/s 2023-10-05 02:47:21 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.1kiB/s ⬆ 0.1kiB/s 2023-10-05 02:47:26 [Relaychain] ⚙️ Syncing 33.2 bps, target=#17583143 (40 peers), best: #8275791 (0x0547…9a7e), finalized #8275456 (0xf374…335f), ⬇ 329.8kiB/s ⬆ 319.8kiB/s 2023-10-05 02:47:26 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 17.8kiB/s ⬆ 0.2kiB/s 2023-10-05 02:47:31 [Relaychain] ⚙️ Syncing 33.2 bps, target=#17583144 (40 peers), best: #8275957 (0x24ea…4287), finalized #8275456 (0xf374…335f), ⬇ 396.9kiB/s ⬆ 320.8kiB/s 2023-10-05 02:47:31 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 0.5kiB/s ⬆ 0.5kiB/s 2023-10-05 02:47:36 [Relaychain] ⚙️ Syncing 10.8 bps, target=#17583144 (40 peers), best: #8276011 (0x499e…7ad2), finalized #8276005 (0xc890…fa1d), ⬇ 326.3kiB/s ⬆ 308.0kiB/s 2023-10-05 02:47:36 [Parachain] 💤 Idle (11 peers), best: #4600769 (0x1134…0c6c), finalized #3481 (0x79ee…c8f0), ⬇ 12 B/s ⬆ 12 B/s

.... .... the RAM usage keeps increasing..... (the capacity is 16GB )

2023-10-05 05:45:07 [Relaychain] ⚙️ Syncing 37.5 bps, target=#17584918 (36 peers), best: #8648566 (0x47ee…85fd), finalized #8648192 (0xd17f…ce76), ⬇ 732.9kiB/s ⬆ 23.1kiB/s 2023-10-05 05:45:08 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 51 B/s ⬆ 51 B/s 2023-10-05 05:45:12 [Relaychain] ⚙️ Syncing 32.3 bps, target=#17584918 (35 peers), best: #8648728 (0x3c2d…347d), finalized #8648704 (0x958b…965a), ⬇ 532.0kiB/s ⬆ 36.0kiB/s 2023-10-05 05:45:13 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 51 B/s ⬆ 51 B/s 2023-10-05 05:45:17 [Relaychain] ⚙️ Syncing 35.7 bps, target=#17584919 (39 peers), best: #8648907 (0x80e7…1ab7), finalized #8648704 (0x958b…965a), ⬇ 881.9kiB/s ⬆ 37.2kiB/s 2023-10-05 05:45:18 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 23.8kiB/s ⬆ 0.3kiB/s 2023-10-05 05:45:22 [Relaychain] ⚙️ Syncing 37.2 bps, target=#17584920 (37 peers), best: #8649093 (0x6140…4b83), finalized #8648704 (0x958b…965a), ⬇ 618.9kiB/s ⬆ 29.2kiB/s 2023-10-05 05:45:23 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 51 B/s ⬆ 51 B/s 2023-10-05 05:45:27 [Relaychain] ⚙️ Syncing 30.2 bps, target=#17584921 (30 peers), best: #8649244 (0xc189…7beb), finalized #8649216 (0x6017…599c), ⬇ 528.7kiB/s ⬆ 37.7kiB/s 2023-10-05 05:45:28 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 18.1kiB/s ⬆ 0.2kiB/s 2023-10-05 05:45:32 [Relaychain] ⚙️ Syncing 35.5 bps, target=#17584922 (31 peers), best: #8649422 (0xde64…6078), finalized #8649216 (0x6017…599c), ⬇ 1023.3kiB/s ⬆ 47.8kiB/s 2023-10-05 05:45:33 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 70 B/s ⬆ 64 B/s 2023-10-05 05:45:37 [Relaychain] ⚙️ Syncing 37.1 bps, target=#17584923 (30 peers), best: #8649608 (0xd02a…1bdb), finalized #8649216 (0x6017…599c), ⬇ 535.2kiB/s ⬆ 43.8kiB/s 2023-10-05 05:45:38 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 18.7kiB/s ⬆ 0.7kiB/s 2023-10-05 05:45:42 [Relaychain] ⚙️ Syncing 29.3 bps, target=#17584923 (35 peers), best: #8649755 (0xd6b5…eaff), finalized #8649728 (0x6963…9fab), ⬇ 741.0kiB/s ⬆ 25.1kiB/s 2023-10-05 05:45:43 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 44 B/s ⬆ 44 B/s 2023-10-05 05:45:47 [Relaychain] ⚙️ Syncing 36.1 bps, target=#17584924 (32 peers), best: #8649936 (0x2e21…4b3a), finalized #8649728 (0x6963…9fab), ⬇ 1008.9kiB/s ⬆ 44.4kiB/s 2023-10-05 05:45:48 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 70 B/s ⬆ 70 B/s 2023-10-05 05:45:52 [Relaychain] ⚙️ Syncing 36.3 bps, target=#17584925 (32 peers), best: #8650118 (0x5810…bcba), finalized #8649728 (0x6963…9fab), ⬇ 538.1kiB/s ⬆ 41.3kiB/s 2023-10-05 05:45:53 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 35.0kiB/s ⬆ 0.4kiB/s 2023-10-05 05:45:57 [Relaychain] ⚙️ Syncing 24.3 bps, target=#17584926 (36 peers), best: #8650240 (0x717c…1837), finalized #8649728 (0x6963…9fab), ⬇ 543.7kiB/s ⬆ 38.4kiB/s 2023-10-05 05:45:58 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 1.2kiB/s ⬆ 1.1kiB/s 2023-10-05 05:46:02 [Relaychain] ⚙️ Syncing 5.1 bps, target=#17584927 (35 peers), best: #8650266 (0x212c…6a77), finalized #8650256 (0xa7a7…c90a), ⬇ 332.9kiB/s ⬆ 44.3kiB/s 2023-10-05 05:46:03 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 57.2kiB/s ⬆ 3.3kiB/s 2023-10-05 05:46:07 [Relaychain] ⚙️ Syncing 6.5 bps, target=#17584928 (37 peers), best: #8650299 (0x206e…0382), finalized #8650256 (0xa7a7…c90a), ⬇ 123.3kiB/s ⬆ 60.3kiB/s 2023-10-05 05:46:08 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 0.2kiB/s ⬆ 0.2kiB/s 2023-10-05 05:46:12 [Relaychain] ⚙️ Syncing 28.7 bps, target=#17584928 (40 peers), best: #8650443 (0x993e…bad7), finalized #8650256 (0xa7a7…c90a), ⬇ 599.3kiB/s ⬆ 53.4kiB/s 2023-10-05 05:46:13 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 6.3kiB/s ⬆ 2.0kiB/s 2023-10-05 05:46:17 [Relaychain] ⚙️ Syncing 13.1 bps, target=#17584929 (40 peers), best: #8650509 (0x224d…cbd2), finalized #8650256 (0xa7a7…c90a), ⬇ 501.6kiB/s ⬆ 56.0kiB/s 2023-10-05 05:46:20 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 10.7kiB/s ⬆ 0.4kiB/s 2023-10-05 05:46:22 [Relaychain] ⚙️ Syncing 3.1 bps, target=#17584929 (40 peers), best: #8650525 (0x2ea6…487b), finalized #8650256 (0xa7a7…c90a), ⬇ 59.3kiB/s ⬆ 36.9kiB/s 2023-10-05 05:46:25 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 76 B/s ⬆ 70 B/s 2023-10-05 05:46:35 [Relaychain] ⚙️ Syncing 1.2 bps, target=#17584931 (38 peers), best: #8650541 (0x9912…7c21), finalized #8650256 (0xa7a7…c90a), ⬇ 17.4kiB/s ⬆ 9.9kiB/s 2023-10-05 05:46:35 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 1.0kiB/s ⬆ 0.2kiB/s 2023-10-05 05:46:40 [Relaychain] ⚙️ Syncing 13.3 bps, target=#17584933 (40 peers), best: #8650608 (0x3629…7e29), finalized #8650256 (0xa7a7…c90a), ⬇ 346.5kiB/s ⬆ 92.9kiB/s 2023-10-05 05:46:40 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 53.6kiB/s ⬆ 1.2kiB/s 2023-10-05 05:46:45 [Parachain] 💤 Idle (13 peers), best: #4600769 (0x1134…0c6c), finalized #68325 (0x5182…cb09), ⬇ 76 B/s ⬆ 76 B/s 2023-10-05 05:46:45 [Relaychain] ⚙️ Syncing 23.7 bps, target=#17584934 (38 peers), best: #8650727 (0xd3c8…29de), finalized #8650256 (0xa7a7…c90a), ⬇ 473.6kiB/s ⬆ 11.1kiB/s [11719.707590] Out of memory: killed process 1196 (acala) total-vm:641883904KB, anon-rss:15479608KB, file-rss:0KB, shmem-rss:0KB, UID:1000pgtables:52380KB, oom_score_adj:0

[xlc]

Can you try a different server? Your report looks similar to this one https://github.com/AcalaNetwork/Acala/issues/2576 and it is the server issue.

[SixoneKui]

Thanks, I will try. Now I am using Hetzner Cloud .

[SixoneKui]

this memory issue seems still there with other Sever (digital ocean) every time the process reaches 8551424 then the ram usages goes high, above 16GB, and then the procedure is killed. I did not run it on a mounted disk. This is the command i used. docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933

23.3kiB/s ⬆ 0.4kiB/s
2023-11-04 02:35:04 [Relaychain] ⚙️ Syncing 0.0 bps, target=#18014281 (32 peers), best: #8551424 (0x8867…262f), finalized #8550400 (0x940d…516a), ⬇ 51.3kiB/s ⬆ 23.8kiB/s
2023-11-04 02:35:04 [Parachain] 💤 Idle (9 peers), best: #4815393 (0x31d5…5d57), finalized #31174 (0x9caf…18b8), ⬇ 0.3kiB/s ⬆ 0.3kiB/s
2023-11-04 02:35:07 [Parachain] ✨ Imported #4815397 (0x096e…59e9)
2023-11-04 02:35:07 [Parachain] ✨ Imported #4815397 (0x793c…ce3d)
2023-11-04 02:35:09 [Relaychain] ⚙️ Syncing 0.0 bps, target=#18014283 (38 peers), best: #8551424 (0x8867…262f), finalized #8550400 (0x940d…516a), ⬇ 59.7kiB/s ⬆ 28.7kiB/s
2023-11-04 02:35:09 [Parachain] 💤 Idle (10 peers), best: #4815393 (0x31d5…5d57), finalized #31174 (0x9caf…18b8), ⬇ 21.4kiB/s ⬆ 9.6kiB/s
2023-11-04 02:35:14 [Relaychain] ⚙️ Syncing 0.0 bps, target=#18014285 (36 peers), best: #8551424 (0x8867…262f), finalized #8550400 (0x940d…516a), ⬇ 82.2kiB/s ⬆ 26.3kiB/s
2023-11-04 02:35:14 [Parachain] 💤 Idle (10 peers), best: #4815393 (0x31d5…5d57), finalized #31174 (0x9caf…18b8), ⬇ 84 B/s ⬆ 88 B/s

info:

2023-11-04 02:30:22 Acala Node 2023-11-04 02:30:22 ✌️ version 2.20.0-unknown
2023-11-04 02:30:22 ❤️ by Acala Developers, 2019-2023 2023-11-04 02:30:22 📋 Chain specification: Acala 2023-11-04 02:30:22 🏷 Node name: aaaa 2023-11-04 02:30:22 👤 Role: FULL 2023-11-04 02:30:22 💾 Database: RocksDb at /acala/data/chains/acala/db/full 2023-11-04 02:30:22 Parachain id: Id(2000) 2023-11-04 02:30:22 Is collating: no 2023-11-04 02:34:18 [Relaychain] 🏷 Local node identity is: 12D3KooWC7TL1xHcXm9kDmRH4taWCqXkA8XrRe15sjcNxtDLHFuv

2023-11-04 02:34:19 [Relaychain] 💻 Operating system: linux 2023-11-04 02:34:19 [Relaychain] 💻 CPU architecture: x86_64 2023-11-04 02:34:19 [Relaychain] 💻 Target environment: gnu 2023-11-04 02:34:19 [Relaychain] 💻 CPU: DO-Premium-Intel 2023-11-04 02:34:19 [Relaychain] 💻 CPU cores: 8 2023-11-04 02:34:19 [Relaychain] 💻 Memory: 15983MB 2023-11-04 02:34:19 [Relaychain] 💻 Kernel: 6.5.0-9-generic 2023-11-04 02:34:19 [Relaychain] 💻 Linux distribution: Ubuntu 20.04.4 LTS 2023-11-04 02:34:19 [Relaychain] 💻 Virtual machine: yes 2023-11-04 02:34:19 [Relaychain] 📦 Highest known block at #8550912 2023-11-04 02:34:19 [Relaychain] Running JSON-RPC server: addr=127.0.0.1:9945, allowed origins=["http://localhost :", "http://127.0.0.1:", "https://localhost:", "https://127.0.0.1:", "https://polkadot.js.org"] 2023-11-04 02:34:19 [Relaychain] 〽️ Prometheus exporter started at 127.0.0.1:9616 2023-11-04 02:34:19 [Relaychain] Starting with an empty approval vote DB. 2023-11-04 02:34:19 [Parachain] 🏷 Local node identity is: 12D3KooWMTYcaYnW4QiY8fjev87LUXT6j4abZ9yPNSy8rhARrg1r

2023-11-04 02:34:19 [Parachain] 💻 Operating system: linux 2023-11-04 02:34:19 [Parachain] 💻 CPU architecture: x86_64 2023-11-04 02:34:19 [Parachain] 💻 Target environment: gnu 2023-11-04 02:34:19 [Parachain] 💻 CPU: DO-Premium-Intel

Could you please check the reason ?

[xlc]

If you are not build an archive mode, I will suggest enable wrap sync

docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

[SixoneKui]

If you are not build an archive mode, I will suggest enable wrap sync

docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

thanks I just tried. Let's see how it works

docker run -it --privileged=true acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

"-- --sync=warp" is part of the command, right? thanks again

[xlc]

yes. the flags before double dash is for parachain and after is for relaychain. in this case, we want to enable warp sync for both relaychain and parachain and therefore two of it

[SixoneKui]

Hi, this command works. Thank you.

But I met another problem. I tried the command below. I remember --ws-port=9944 works before. But it now gives "error: unexpected argument '--ws-external' found"

root@acala-2:~/kkc/acala# sudo docker run -it --privileged=true -v /mnt/volume_fra1_01:/acala/data acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --ws-external --rpc-external --rpc-cors=all --ws-port=9944 --rpc-port=9933 --sync=warp -- --sync=warp error: unexpected argument '--ws-external' found

tip: a similar argument exists: '--rpc-external'

Usage: acala --name --rpc-external <--chain |--dev|--base-path |--log ...|--detailed-log-output|--disable-log-color|--enable-log-reloading|--tracing-targets |--tracing-receiver > [-- ...]

For more information, try '--help'. root@acala-2:~/kkc/acala#

Now I have a node running with

sudo docker run -it --privileged=true -v /mnt/volume_fra1_01:/acala/data acala/acala-node:2.20.0 --base-path=/acala/data --chain=acala --name=aaaa --rpc-external --rpc-cors=all --rpc-port=9933 --sync=warp -- --sync=warp

I tried ws://172.17.0.2:9944 to connect but it does not work.

How do I connect to WS?

THank you again.

[xlc]

the http port and ws port are merged so you only need one specified with --rpc-port. please check the release note https://github.com/AcalaNetwork/Acala/releases/tag/2.20.0

[qezz]

We also observe an OOM while syncing a node.

More over, the acala's finalized block went to 0, and its catching up speed doesn't look good.

We run it as follows:

ExecStart=/opt/validators/acala/bin/2.24.0/acala \
        --base-path=/opt/validators/acala/data \
        --chain=acala \
        --name=chorusone-rpc-1 \
        --pruning=archive \
# workaround for substrate bug, see https://github.com/paritytech/substrate/issues/9697
        --state-cache-size 0 \
        --rpc-external \
        --rpc-cors=all \
        --rpc-port=9944 \
        --rpc-max-connections=2000 \
        --execution=wasm \
        --prometheus-port 10140 \
        --prometheus-external \
        --listen-addr=/ip4/0.0.0.0/tcp/30333 \
        --listen-addr=/ip4/0.0.0.0/tcp/30335 \
        -- \
        --chain=polkadot \
        --prometheus-external \
        --prometheus-port 9616

We also run the eth-rpc-adapter alongside, but I believe since Relaychain didn't sync yet, it wasn't doing anything useful.

[xlc]

@qezz it is expected the finalized block to be 0 while the relaychain part is syncing can you share some logs

[qezz]

it is expected the finalized block to be 0 while the relaychain part is syncing

It synced from 0 to (current head) in a day, and then after another day dropped to 0. Is this expected?

can you share some logs

Will the logs for the OOM loop work? I stopped the node for now, so it doesn't cause issues for collocated processes. It was OOMing in a loop within ~1h, i.e. it crashed on OOM, restarted, and then reached OOM within an hour, and restarted again etc. etc. I don't think I have logs before that, as they are probably rotated by this point.

[qezz]

it is expected the finalized block to be 0 while the relaychain part is syncing It synced from 0 to (current head) in a day, and then after another day dropped to 0. Is this expected?

Sorry, I'm confusing things here. I mean finalized height for Parachain, and that's what shown on the screenshot

Edit:

Relaychain's finalized height was indeed 0, and that's OK. It has started growing after ~2 days of syncing, and then it went OOM after several hours

[xlc]

ok it is not expected for the parachain finalized height goes from a non zero value to zero

Will the logs for the OOM loop work?

that will be useful

[SixoneKui]

Hi,

I tried to enable wrap sync with: --sync=warp -- --sync=warp But I see below

Error: Service(Other("Warp sync enabled, but no warp sync provider configured."))

how to solve this ? or is the sync provider configured on my side ?

(this is not form acala node but from parallel, could please also give some clues)

Many thanks

[xlc]

@SixoneKui this is not related to the original issue and it is not even related to this repo so please avoid posting unrelated content in the issue next time.

for this particular question, my guess is that warp sync is not enabled in the client but I did not check their code and can't really tell

[qezz]

The logs are basically empty on the OOM loop

# journalctl -alxfu acala --no-hostname
-- Logs begin at Tue 2024-04-30 04:06:54 UTC. --
Apr 30 12:53:52 acala[933902]: CLI parameter `--execution` has no effect anymore and will be removed in the future!
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 Acala Node
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 ✌️  version 2.24.0-unknown
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 ❤️  by Acala Developers, 2019-2024
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 📋 Chain specification: Acala
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 🏷  Node name: chorusone-rpc-1
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 👤 Role: FULL
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 💾 Database: RocksDb at /opt/validators/acala/data/chains/acala/db/full
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 Parachain id: Id(2000)
Apr 30 12:53:52 acala[933902]: 2024-04-30 12:53:52 Is collating: no

no more logs after this ^, and it's already going OOM

[xlc]

This issue is very unlikely caused by Acala specific code as we don't really have any non standard code compare to other parachains.

I've made a new release branch prepare for 2.25.0 release: https://github.com/AcalaNetwork/Acala/tree/release-acala-2.25.0 Could you give it a try and see if it helps? It includes a new version of polkadot-sdk.

The docker image can be found here: https://github.com/AcalaNetwork/Acala/pkgs/container/acala-node/210655705?tag=2.25.0

[bkchr]

Please also run with -ltrace to get more logs.

[qezz]

I didn't manage to make it sync from genesis. The solution for me was to rsync a datadir from one node to another