CKB Process Occasionally Fails to Terminate

sunchengzhu commented 3 months ago

Bug Report

Current Behavior

I restart CKB every two hours and I've noticed that there are occasional issues with missing metrics data.

Terminate command:

killckb() {
    PIDS=$(sudo lsof -ti:8114)
    for i in $PIDS; do
        echo "killed the ckb $i"
        sudo kill $i
    done
}

killckb

Expected Behavior

metrics data is not lost

Environment

CKB version: ckb 0.118.0-rc1 (63ae338 2024-08-19)
Chain: mainnet
Operating system: Ubuntu 20.04.6 LTS
Arch: x64

Additional context/Screenshots

My ckb.toml configuration:

[metrics.exporter.prometheus]
target = { type = "prometheus", listen_address = "0.0.0.0:8100" }

# # Experimental: Monitor memory changes.
[memory_tracker]
# # Seconds between checking the process, 0 is disable, default is 0.
interval = 5

15168316096 commented 3 months ago

This is an intermittent issue.

The situation you described indicates that other nodes running ckb v118 did not experience missing metrics during the same time period. If the data were not collected from the source, it would suggest that the ckb process was terminated. However, based on the ckb logs we reviewed earlier, it shows that the service restarted successfully and continued to sync normally.

Therefore, it is more likely that the issue occurred when Prometheus was scraping data from ckb.

sunchengzhu commented 3 months ago

From the run.log, it can be seen that the CKB process did not exit promptly:

Received the Ctrl-C signal at 15:20.
At 15:25, an attempt to start CKB resulted in an ERROR.
The CKB shutdown log was not seen until 15:26.

sunchengzhu commented 3 months ago

This issue has been verified and resolved in PR 4615.

nervosnetwork / ckb