search

nervosnetwork / ckb

The Nervos CKB is a public permissionless blockchain, and the layer 1 of Nervos network.
https://www.nervos.org
MIT License
1.15k stars 228 forks source link

Resource utilization of CKB node server. #4078

Closed guomaoqiu closed 1 month ago

guomaoqiu commented 1 year ago

Hi, I need some help with the following questions:

The memory usage of the CKB node server keeps increasing after running for a period of time, although the increase rate is not very noticeable. However, this is a potentially dangerous sign, as there are other services running on the server as well.

  1. Is there no memory release mechanism for this service?
  2. If I want to limit its resources, what would be a reasonable allocation of memory/CPU?
  3. Apart from restarting the service to release memory, is there any other way to handle this issue?
image
eval-exec commented 1 year ago

Hello, What's the version of the CKB node? And how did you run the CKB node? Which OS is the CKB node running on? Which tool do you use to report the CPU/MEMORY usage? Is public-testnet-ckb-0 the process name of ckb, or is it the hostname of your machine? Is 6508Mi the sum of all the processes' memory on your machine? What does 50m below the CPU(cores) field mean?

Could you please show the output of the htop, free -h, and hostname commands?

guomaoqiu commented 1 year ago

Hello, What's the version of the CKB node? And how did you run the CKB node? Which OS is the CKB node running on? Which tool do you use to report the CPU/MEMORY usage? Is public-testnet-ckb-0 the process name of ckb, or is it the hostname of your machine? Is 6508Mi the sum of all the processes' memory on your machine? What does 50m below the CPU(cores) field mean?

Could you please show the output of the htop, free -h, and hostname commands?

  1. It is a container pod running in Kubernetes, named "public-testnet-ckb-0".
  2. The operating system of the host node for Kubernetes is Ubuntu 22.04.1.
  3. The container image version of ckb is "nervos/ckb:v0.110.0".
  4. The screenshot above shows the result obtained by using the command "kubectl top pod". ":50m" means "50 millicores", which represents 50/1000 of a core.
  5. "6508Mi represents the working set size. This is an explanation for monitoring: 
    
RSS and WSS are metrics used to represent the memory usage of a process or container. The differences between them are as follows:

RSS (Resident Set Size): Represents the actual physical memory size used by the process, including code, data, heap, stack, and shared libraries. However, RSS does not include memory swapped to disk, so it cannot accurately reflect the actual memory usage of a process.

WSS (Working Set Size): Represents the current memory size being used by the process, including everything in RSS as well as files and caches used by the process. WSS is typically used to measure the actual memory needs of a process for memory allocation and optimization.

Therefore, WSS can more accurately reflect the actual memory usage of a process, but it is typically larger than RSS and changes with the process's running state. In container environments, the 'container_memory_working_set_bytes' metric is typically used to measure the WSS of a container."


6. The following is the status of this pod running on the host machine: 
![image](https://github.com/nervosnetwork/ckb/assets/9714859/b437dac5-4492-43d6-988e-19434403035a)
![image](https://github.com/nervosnetwork/ckb/assets/9714859/3a040683-c8f1-407b-a26a-1bce0190fbfd)
![image](https://github.com/nervosnetwork/ckb/assets/9714859/82f48a62-6bcc-4a43-89b7-6a82ac0438aa)
quake commented 1 year ago

Is your ckb synchronized from height 0, if so, according to your screenshot, it used rss is about 2G, this is normal, generally synchronized from 0, according to different environment, the maximum rss memory will be used between 2.5 ~ 3G.

guomaoqiu commented 1 year ago

It's not syncing from 0, it has been running normally for a long time. I'm just wondering why it's using more resources. As you can see from the figure above, the memory usage has been consistently above 4GB.

"It's not syncing from 0, it has been running normally for a long time. I'm just wondering why it's using more resources. As you can see from the figure above, the memory usage has been consistently above 4GB."

eval-exec commented 1 year ago

I suggest to monitor both of container_memory_working_set_bytes and container_memory_rss metrics.

Could you please show the graph of container_memory_rss?

guomaoqiu commented 1 year ago

I suggest to monitor both of container_memory_working_set_bytes and container_memory_rss metrics.

Could you please show the graph of container_memory_rss?

image

eval-exec commented 1 year ago

Could you please scale the graph's time range to 1 day, 1 week and 1 month?

guomaoqiu commented 1 year ago

Could you please scale the graph's time range to 1 day, 1 week and 1 month?

image image image （We currently only keep monitoring data for 15 days.)

eval-exec commented 1 year ago

In the recent 15 days, the container_memory_rss metric is abot 2.0 ~ 2.3G, and did not exceed 3G, indicating it's currently in a safe range.

(And I read this: memory measurements on kubernets , learned that "active cache is counted as part of the working set container_memory_working_set_bytes. And kubernets will evicting the pod when under pressure."

Emm. I recommend setting up an alert to be triggered when the container_memory_rss reaches 6G or container_memory_working_set_bytes reaches 12G.)

guomaoqiu commented 1 year ago

In the recent 15 days, the container_memory_rss metric is abot 2.0 ~ 2.3G, and did not exceed 3G, indicating it's currently in a safe range.

(And I read this: memory measurements on kubernets , learned that "active cache is counted as part of the working set container_memory_working_set_bytes. And kubernets will evicting the pod when under pressure."

Emm. I recommend setting up an alert to be triggered when the container_memory_rss reaches 6G or container_memory_working_set_bytes reaches 12G.)

Okay, so currently it is occupying memory and I want to free up some memory usage. Is the only way to do this by restarting it?

quake commented 1 year ago

The memory usage looks normal, the rss is only about 2.0G according to your graph, I don't think you need to restart ckb service.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 1 month ago

This issue was closed because it has been stalled for 5 days with no activity.