jimiflowers
commented
1 year ago UPDATE
New behaviour: NODE1's Phala processes stop all operations and get freezed all at the same time. No visible errors on Node, PRuntime or Headers Cache, only Pherry says "exited with error: Rpc error: Request timeout" and "Restarting..." but does not restart, all containers keep freezed and also prometheus services (9615 and 9616) stop responding.. Attached logs from all containers for this session (I have had to supress some intermediate lines from PRuntime log to fit the github files max size of 25MB).
Docker status for all containers once they get freezed:
root@khalanuc ~ # docker inspect -f '{{json .State}}' "phala-node"
{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":47521,"ExitCode":0,"Error":"","StartedAt":"2022-12-12T10:31:20.754943891Z","FinishedAt":"0001-01-01T00:00:00Z"}
root@khalanuc ~ # docker inspect -f '{{json .State}}' "phala-pherry"
{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":48161,"ExitCode":0,"Error":"","StartedAt":"2022-12-12T10:32:08.527511555Z","FinishedAt":"2022-12-12T10:32:07.550963243Z"}
root@khalanuc ~ # docker inspect -f '{{json .State}}' "phala-pruntime"
{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":47353,"ExitCode":0,"Error":"","StartedAt":"2022-12-12T10:31:20.481902779Z","FinishedAt":"0001-01-01T00:00:00Z"}
root@khalanuc ~ # docker inspect -f '{{json .State}}' "phala-headers-cache"
{"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":47295,"ExitCode":0,"Error":"","StartedAt":"2022-12-12T10:31:20.613639649Z","FinishedAt":"0001-01-01T00:00:00Z"}root@khalanuc ~ # docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a653725461b2 phalanetwork/phala-pherry "/root/pherry -r --p…" 10 hours ago Up 10 hours phala-pherry
f9458ca4cb5f phalanetwork/khala-node "/usr/bin/tini -- /b…" 10 hours ago Up 10 hours 0.0.0.0:9615-9616->9615-9616/tcp, :::9615-9616->9615-9616/tcp, 0.0.0.0:9933-9934->9933-9934/tcp, :::9933-9934->9933-9934/tcp, 0.0.0.0:9944-9945->9944-9945/tcp, :::9944-9945->9944-9945/tcp, 0.0.0.0:31333-31334->31333-31334/tcp, :::31333-31334->31333-31334/tcp, 30333-30334/tcp phala-node
14256f1c8452 phalanetwork/phala-pruntime "/usr/bin/tini -- /b…" 10 hours ago Up 10 hours 0.0.0.0:8000->8000/tcp, :::8000->8000/tcp phala-pruntime
3ac76203cc73 phalanetwork/phala-headers-cache "/opt/headers-cache/…" 10 hours ago Up 10 hours 0.0.0.0:8002->8002/tcp, :::8002->8002/tcp phala-headers-cache
root@khalanuc ~ #After restart all processes I have seen this error repeated 64 times in Node logs and these "warnings", KSM chain works normally but the Node cannot import Khala blocks, and Pherry gives Network connection errors again:
phala-node | 2022-12-12 20:24:41 [Parachain] panicked at 'Bad input data provided to validate_transaction: Codec error', /home/h4x/workspace/khala-parachain/runtime/khala/src/lib.rs:1594:1phala-node | 2022-12-12 20:43:52 ⚠️ System declares internal migrations (which *might* execute). On-chain `<wasm:stripped>` vs current storage version `<wasm:stripped>`
phala-node | 2022-12-12 20:43:52 ⚠️ PhalaStakePool declares internal migrations (which *might* execute). On-chain `<wasm:stripped>` vs current storage version `<wasm:stripped>`
phala-node | 2022-12-12 20:43:52 ⚠️ PhalaMining declares internal migrations (which *might* execute). On-chain `<wasm:stripped>` vs current storage version `<wasm:stripped>`
phala-node | 2022-12-12 20:43:52 ⚠️ PhalaRegistry declares internal migrations (which *might* execute). On-chain `<wasm:stripped>` vs current storage version `<wasm:stripped>`And Pherry reports:
phala-pherry | [2022-12-12T20:54:28.659660Z INFO jsonrpsee_client_transport::ws] Connection established to target: Target { sockaddrs: [], host: "phala-node", host_header: "phala-node:9945", _mode: Plain, path_and_query: "/" }
phala-pherry | [2022-12-12T20:54:28.734444Z INFO pherry] Connected to relaychain at: ws://phala-node:9945
phala-pherry | [2022-12-12T20:54:28.738270Z INFO jsonrpsee_client_transport::ws] Connection established to target: Target { sockaddrs: [], host: "phala-node", host_header: "phala-node:9944", _mode: Plain, path_and_query: "/" }
phala-pherry | [2022-12-12T20:54:28.772612Z INFO pherry] Connected to parachain node at: ws://phala-node:9944
phala-pherry | [2022-12-12T20:54:28.788324Z INFO pherry] bridge() exited with error: error sending request for url (http://phala-pruntime:8000/prpc/PhactoryAPI.GetInfo): error trying to connect: tcp connect error: Connection refused (os error 111)
phala-pherry | [2022-12-12T20:54:30.790581Z INFO pherry] Restarting...I needed to restart few times (5 times) to get Node working right again, importing KSM blocks and Khala blocks. All is working fine for 3 or 4 minutes, and then all containers get freezed again. No matter how many times I restart the processes, there is no way for the node to import blocks from the khala network continuously.
jasl
I have two nodes, both with same hardware (Intel NUC7PJYH2) and both of them suffer the same behaviour.
Both nodes get stuck at a block height (Khala network, KSM syncs well) and Pherry restarts because it fails to connect to node. It's very random, cos sometimes I restart the processes and all works fine during several hours (12h o 14h) but sometimes I need to restart processes 2, 3 or 4 times until I get all working fine. I have tried reinstalling both nodes from scratch, using databases synced from zero, with and without KSM snapshot and even copying databases from other nodes that are running fine.
NODE1: It has been running smoothly for months, but after a coolingdown period, begins to show the issue. NODE2: It hasn't been running ever, it's a brand new machine installed for the first time.
I have attached a file with logs (from processes start until the fail) and system info from both nodes. Please, don't hesitate to ask for any further info you need.
nodes_info_and_logs.tar.gz