Open SuryaSHalwasia opened 2 months ago
It doesn't look like the node(s) are crashing. It looks more like the indy-node-control
service is restarting them (possibly due to performance while under load). The indy-node-control
service does not run in the docker containers.
Could you please provide the following information (redact any private info):
/var/log/indy/<network_name>/<node_alias>.log
/var/log/indy/node_control.log
sudo journalctl -u indy-node
sudo journalctl -u indy-node-control
sudo validator-info -v
cat /etc/indy/indy_config.py
cat /etc/indy/indy.env
cat /etc/indy/node_control.conf
node inbox size : 0 client inbox size : 0 age (seconds) : 352.6427562236786 next check for reconnect: 2.526324950000344 node connections : {'Node2', 'Node3', 'Node4'} f : 1 master instance : 0 replicas : 2 view no : 0 rank : 0 msgs to replicas : 0 action queue : 0 139717908141536 action queue stash : 5 139717908141632 2024-07-07 23:36:35,646|INFO|node.py|Node1 reseting... 2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1 closing its listener 2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1 stopped 2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1C closing its listener 2024-07-07 23:36:35,664|NOTIFICATION|zstack.py|stack Node1C stopped 2024-07-07 23:36:35,665|INFO|node.py|Node1 closing key-value storages 2024-07-07 23:36:36,658|INFO|motor.py|Node1 changing status from stopping to stopped 2024-07-07 23:36:36,658|NOTIFICATION|looper.py|Looper shut down in 1.012 seconds.
In others it is 2024-07-07 23:36:35,767|NOTIFICATION|keep_in_touch.py|Node3's connections changed from {'Node2', 'Node4', 'Node1'} to {'Node2', 'Node4'} 2024-07-07 23:36:35,767|NOTIFICATION|keep_in_touch.py|CONNECTION: Node3 disconnected from Node1
2024-07-07 23:36:35,767|INFO|motor.py|Node3 changing status from started to started_hungry 2024-07-07 23:36:35,767|NOTIFICATION|primary_connection_monitor_service.py|Node3:0 lost connection to primary of master 2024-07-07 23:36:35,767|INFO|primary_connection_monitor_service.py|Node3:0 scheduling primary connection check in 60 sec
In /var/log/indy/node_control.log (This was only present in Node1) 2024-06-26 20:15:24,574|INFO|node_control_tool.py|Node control tool is starting up on localhost port 30003 2024-06-26 20:15:25,142|INFO|node_control_utils.py|Successfully put indy-plenum indy-node python3-ioflo python3-orderedset python3-base58 python3-prompt-toolkit python3-rlp python3-sha3 python3-libnacl python3-six python3-portalocker python3-sortedcontainers python3-dateutil python3-semver python3-psutil python3-pyzmq python3-jsonpickle python3-rocksdb python3-pympler python3-packaging python3-timeout-decorator python3-distro packages on hold 2024-06-26 20:15:25,142|DEBUG|node_control_tool.py|Waiting for the next event
For journalctl -u indy-node -- Logs begin at Sat 2023-12-02 18:58:18 UTC, end at Mon 2024-07-08 14:56:20 UTC. -- -- No entries --
For indy-node-control
-- Logs begin at Sat 2023-12-02 18:58:18 UTC, end at Mon 2024-07-08 14:58:55 UTC. -- Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Started Service for upgrade of existing Indy Node and other operations. Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: indy-node-control.service: Succeeded. Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations. Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Started Service for upgrade of existing Indy Node and other operations. Jun 26 20:15:24 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:24,574|INFO|node_control_tool.py|Node control tool is starting up o> Jun 26 20:15:25 ip-172-31-40-65 env[19732]: indy-plenum set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: indy-node set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-ioflo set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-orderedset set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-base58 set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-prompt-toolkit set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-rlp set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-sha3 set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-libnacl set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-six set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-portalocker set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-sortedcontainers set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-dateutil set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-semver set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-psutil set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-pyzmq set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-jsonpickle set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-rocksdb set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-pympler set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-packaging set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-timeout-decorator set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-distro set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:25,142|INFO|node_control_utils.py|Successfully put indy-plenum indy> Jun 26 20:15:25 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:25,142|DEBUG|node_control_tool.py|Waiting for the next event Jun 27 04:22:39 ip-172-31-40-65 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Jun 27 04:22:40 ip-172-31-40-65 systemd[1]: indy-node-control.service: Succeeded. Jun 27 04:22:40 ip-172-31-40-65 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations.
For validator-info "Hardware": "HDD_all": 6405 Mbs "RAM_all_free": 31119 Mbs "RAM_used_by_node": 687 Mbs "Node_info": "BLS_key": 4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba "Catchup_status": "Last_txn_3PC_keys": "0": "Node2": None None "Node3": None None "1": "Node2": None None "Node3": None None "2": "Node2": None None "Node3": None None "3": "Node2": None None "Node3": None None "Ledger_statuses": "0": synced "1": synced "2": synced "3": synced "Number_txns_in_catchup": "0": 0 "1": 0 "2": 0 "3": 0 "Received_LedgerStatus": "Waiting_consistency_proof_msgs": "0": None "1": None "2": None "3": None "Client_ip": 0.0.0.0 "Client_port": 9702 "Client_protocol": tcp "Committed_ledger_root_hashes": "0": b'2PZguAeNfTpfPsavVTfAq98UC89Ko4DgbBa2CpTL1Svv' "1": b'HgxLc6h6R12P5zSM1h7T5jSttmAoHryhmrshffmM9imx' "2": b'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn' "3": b'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn' "Committed_state_root_hashes": "0": b'EQUSi2crBb6G82h8NdUjU8MBNQaHTxRtwvtu9Qi1PAG' "1": b'39aVhSB3idDKLkKTUGfKHFMhE2oNZ5GyCSDqCg4M5x2U' "2": b'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA' "Count_of_replicas": 2 "Freshness_status": "0": "Has_write_consensus": True "Last_updated_time": 2024-07-07 23:30:43+00:00 "1": "Has_write_consensus": True "Last_updated_time": 2024-07-07 23:30:43+00:00 "2": "Has_write_consensus": True "Last_updated_time": 2024-07-07 23:30:43+00:00 "Metrics": "Delta": 0.1 "Lambda": 240 "Omega": 20 "average-per-second": "read-transactions": 0.0 "write-transactions": 0.0 "avg backup throughput": 0.0 "client avg request latencies": "0": None "1": None "instances started": "0": 1785.408685558 "1": 1785.410516859 "master throughput": None "master throughput ratio": None "max master request latencies": 0 "ordered request counts": "0": 0 "1": 0 "ordered request durations": "0": 0 "1": 0 "throughput": "0": 0.0 "1": 0.0 "total requests": 0 "transaction-count": "audit": 0 "config": 0 "ledger": 10 "pool": 4 "uptime": 300 "Mode": participating "Name": Node1 "Node_ip": 0.0.0.0 "Node_port": 9701 "Node_protocol": tcp "Replicas_status": "Node1:0": "Last_ordered_3PC": 0 0 "Primary": Node1:0 "Stashed_txns": "Stashed_PrePrepare": 0 "Stashed_checkpoints": 0 "Watermarks": 0:300 "Node1:1": "Last_ordered_3PC": 0 0 "Primary": Node2:1 "Stashed_txns": "Stashed_PrePrepare": 0 "Stashed_checkpoints": 0 "Watermarks": 0:9223372036854775807 "Requests_timeouts": "Ordering_phase_req_timeouts": 0 "Propagates_phase_req_timeouts": 0 "Uncommitted_ledger_root_hashes": "Uncommitted_ledger_txns": "0": "Count": 0 "1": "Count": 0 "2": "Count": 0 "3": "Count": 0 "Uncommitted_state_root_hashes": "0": b'EQUSi2crBb6G82h8NdUjU8MBNQaHTxRtwvtu9Qi1PAG' "1": b'39aVhSB3idDKLkKTUGfKHFMhE2oNZ5GyCSDqCg4M5x2U' "2": b'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA' "View_change_status": "IC_queue": "1": "Voters": "Node2": "reason": 26 "Node3": "reason": 26 "Last_complete_view_no": 0 "Last_view_change_started_at": 1970-01-01 00:00:00 "VCDone_queue": "VC_in_progress": False "View_No": 0 "did": Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv "verkey": 33nHHYKnqmtGAVfZZGoP8hpeExeH45Fo8cKmd5mcnKYk7XgWNBxkkKJ "Pool_info": "Blacklisted_nodes": "Quorums": {'n': 4, 'f': 1, 'weak': Quorum(2), 'strong': Quorum(3), 'propagate': Quorum(2), 'prepare': Quorum(2), 'commit': Quorum(3), 'reply': Quorum(2), 'view_change': Quorum(3), 'election': Quorum(3), 'view_change_ack': Quorum(2), 'view_change_done': Quorum(3), 'same_consistency_proof': Quorum(2), 'consistency_proof': Quorum(2), 'ledger_status': Quorum(2), 'ledger_status_last_3PC': Quorum(2), 'checkpoint': Quorum(2), 'timestamp': Quorum(2), 'bls_signatures': Quorum(3), 'observer_data': Quorum(2), 'backup_instance_faulty': Quorum(2)} "Reachable_nodes": ['Node1', 0] ['Node2', 1] ['Node3', None] ['Node4', None] "Reachable_nodes_count": 4 "Read_only": False "Suspicious_nodes": "Total_nodes_count": 4 "Unreachable_nodes": "Unreachable_nodes_count": 0 "f_value": 1 "Protocol": "Software": "Indy_packages": hi indy-node 1.13.2~rc6 amd64 Indy node hi indy-plenum 1.13.1~rc4 amd64 Plenum Byzantine Fault Tolerant Protocol
"Installed_packages":
ioflo 2.0.2
tzdata 2024.1
distro 1.7.0
indy-plenum 1.13.1rc4
pillow 10.3.0
rlp 0.6.0
fonttools 4.53.0
ujson 1.33
python-rocksdb 0.7.0
importlib_resources 6.4.0
jsonpickle 3.0.1
kiwisolver 1.4.5
importlib-metadata 3.10.1
zipp 3.19.2
cycler 0.12.1
python-ursa 0.1.1
pyzmq 22.3.0
leveldb 0.201
matplotlib 3.7.5
Pympler 0.8
python-dateutil 2.8.2
setuptools 50.3.2
msgpack-python 0.5.6
packaging 21.3
timeout-decorator 0.5.0
psutil 5.9.4
pandas 2.0.3
semver 2.13.0
base58 2.1.1
indy-node 1.13.2rc6
orderedset 2.0.3
sha3 0.2.1
portalocker 2.7.0
libnacl 1.6.1
sortedcontainers 1.5.7
contourpy 1.1.1
prompt-toolkit 3.0.18
numpy 1.24.4
pytz 2024.1
urllib3 1.25.8
unattended-upgrades 0.1
language-selector 0.1
pyrsistent 0.15.5
ubuntu-pro-client 8001
SecretStorage 2.3.1
pyOpenSSL 19.0.0
zipp 1.0.0
netifaces 0.10.4
systemd-python 234
PyJWT 1.7.1
Automat 0.8.0
requests 2.22.0
sos 4.5.6
PyYAML 5.3.1
keyring 18.0.1
more-itertools 4.2.0
command-not-found 0.3
hibagent 1.0.1
jsonpointer 2.0
pyasn1 0.4.2
pluggy 0.13.0
cryptography 2.8
ec2-hibinit-agent 1.0.0
idna 2.8
PyHamcrest 1.9.0
Twisted 18.9.0
six 1.14.0
python-apt 2.0.1+ubuntu0.20.4.1
MarkupSafe 1.1.0
python-debian 0.1.36+ubuntu1.1
wadllib 1.3.3
blinker 1.4
hyperlink 19.0.0
zope.interface 4.7.1
pyasn1-modules 0.2.1
PyNaCl 1.3.0
pexpect 4.6.0
Click 7.0
httplib2 0.14.0
certifi 2019.11.28
jsonpatch 1.22
pyparsing 2.4.6
pyserial 3.4
dbus-python 1.2.16
service-identity 18.1.0
entrypoints 0.3
colorama 0.4.3
oauthlib 3.1.0
chardet 3.0.4
PyGObject 3.36.0
cloud-init 24.1.3
distro-info 0.23+ubuntu1.1
lazr.uri 1.0.3
constantly 15.1.0
attrs 19.3.0
pip 20.0.2
setuptools 45.2.0
ufw 0.36
wheel 0.34.2
jsonschema 3.2.0
Jinja2 2.10.1
ssh-import-id 5.10
requests-unixsocket 0.2.0
launchpadlib 1.10.13
incremental 16.10.1
simplejson 3.16.0
pymacaroons 0.13.0
pytest 4.6.9
atomicwrites 1.1.5
wcwidth 0.1.8
lazr.restfulclient 0.14.2
configobj 5.0.6
py 1.8.1
"OS_version": Linux-5.15.0-1064-aws-x86_64-with-glibc2.29
"indy-node": 1.13.2rc6
"sovrin": unknown
"Update_time": Sunday, July 7, 2024 11:35:43 PM +0000 "response-version": 0.0.1 "timestamp": 1720395343
"Configuration": "Config": "Main_config":
# Disable stdout logging
enableStdOutLogging = True
#logLevel=0
# Directory to store ledger.
LEDGER_DIR = '/var/lib/indy'
# Directory to store logs.
LOG_DIR = '/var/log/indy'
# Directory to store keys.
KEYS_DIR = '/var/lib/indy'
# Directory to store genesis transactions files.
GENESIS_DIR = '/var/lib/indy'
# Directory to store backups.
BACKUP_DIR = '/var/lib/indy/backup'
# Directory to store plugins.
PLUGINS_DIR = '/var/lib/indy/plugins'
# Directory to store node info.
NODE_INFO_DIR = '/var/lib/indy'
NETWORK_NAME = 'sandbox'
"Network_config":
"User_config":
"Genesis_txns":
"domain_txns":
{"reqSignature":{},"txn":{"data":{"dest":"V4SGRU86Z58d6TV7PBUe6f","role":"0","verkey":"~CoRER63DVYnWZtK8uAzNbx"},"metadata":{},"type":"1"},"txnMetadata":{"seqNo":1},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"Th7MpTaRZVRYnPiabds81Y","role":"2","verkey":"~7TYfekw4GUagBnBVCqPjiC"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":2},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"EbP4aYNeTHL6q385GuVpRV","role":"2","verkey":"~RHGNtfvkgPEUQzQNtNxLNu"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":3},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"4cU41vWW82ArfxJxHkzXPG","role":"2","verkey":"~EMoPA6HrpiExVihsVfxD3H"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":4},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"TWwCRQRZ2ZHMJFn9TzLp7W","role":"2","verkey":"~UhP7K35SAXbix1kCQV4Upx"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":5},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"7JhapNNMLnwkbiC2ZmPZSE","verkey":"~LgpYPrzkB6awcHMTPZ9TVn"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":6},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"MEPecrczs4Wh6FA12u519D","verkey":"~A4rMgHYboWYS1DXibCgo9W"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":7},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"EAPtwgevBpzP8hkj9sxuzy","verkey":"~gmzSzu3feXC6g2djF7ar4"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":8},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"LuL1HK1sDruwkfm68jrVfD","verkey":"~Nyv9BKUJuvjgMbfbwk8CFD"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":9},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"dest":"462p8mtcX6jpa9ky565YEL","verkey":"~LCgq4hnSvMvB8nKd9vgsTD"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":10},"ver":"1"}
"pool_txns":
{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node1","blskey":"4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba","blskey_pop":"RahHYiCvoNCtPTrVtP7nMC5eTYrsUA8WjXbdhNc8debh1agE9bGiJxWBXYNFbnJXoXhWFMvyqhqhRoq737YQemH5ik9oL7R4NTTCz2LEZhkgLJzB3QRQqJyBNyv7acbdHrAT8nQ9UkLbaVL9NBpnWXBTw4LEMePaSHEw66RzPNdAX1","client_ip":"172.31.40.65","client_port":9702,"node_ip":"172.31.40.65","node_port":9701,"services":["VALIDATOR"]},"dest":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"metadata":{"from":"Th7MpTaRZVRYnPiabds81Y"},"type":"0"},"txnMetadata":{"seqNo":1,"txnId":"fea82e10e894419fe2bea7d96296a6d46f50f93f9eeda954ec461b2ed2950b62"},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node2","blskey":"37rAPpXVoxzKhz7d9gkUe52XuXryuLXoM6P6LbWDB7LSbG62Lsb33sfG7zqS8TK1MXwuCHj1FKNzVpsnafmqLG1vXN88rt38mNFs9TENzm4QHdBzsvCuoBnPH7rpYYDo9DZNJePaDvRvqJKByCabubJz3XXKbEeshzpz4Ma5QYpJqjk","blskey_pop":"Qr658mWZ2YC8JXGXwMDQTzuZCWF7NK9EwxphGmcBvCh6ybUuLxbG65nsX4JvD4SPNtkJ2w9ug1yLTj6fgmuDg41TgECXjLCij3RMsV8CwewBVgVN67wsA45DFWvqvLtu4rjNnE9JbdFTc1Z4WCPA3Xan44K1HoHAq9EVeaRYs8zoF5","client_ip":"172.31.40.114","client_port":9704,"node_ip":"172.31.40.114","node_port":9703,"services":["VALIDATOR"]},"dest":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"metadata":{"from":"EbP4aYNeTHL6q385GuVpRV"},"type":"0"},"txnMetadata":{"seqNo":2,"txnId":"1ac8aece2a18ced660fef8694b61aac3af08ba875ce3026a160acbc3a3af35fc"},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node3","blskey":"3WFpdbg7C5cnLYZwFZevJqhubkFALBfCBBok15GdrKMUhUjGsk3jV6QKj6MZgEubF7oqCafxNdkm7eswgA4sdKTRc82tLGzZBd6vNqU8dupzup6uYUf32KTHTPQbuUM8Yk4QFXjEf2Usu2TJcNkdgpyeUSX42u5LqdDDpNSWUK5deC5","blskey_pop":"QwDeb2CkNSx6r8QC8vGQK3GRv7Yndn84TGNijX8YXHPiagXajyfTjoR87rXUu4G4QLk2cF8NNyqWiYMus1623dELWwx57rLCFqGh7N4ZRbGDRP4fnVcaKg1BcUxQ866Ven4gw8y4N56S5HzxXNBZtLYmhGHvDtk6PFkFwCvxYrNYjh","client_ip":"172.31.32.116","client_port":9706,"node_ip":"172.31.32.116","node_port":9705,"services":["VALIDATOR"]},"dest":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"metadata":{"from":"4cU41vWW82ArfxJxHkzXPG"},"type":"0"},"txnMetadata":{"seqNo":3,"txnId":"7e9f355dffa78ed24668f0e0e369fd8c224076571c51e2ea8be5f26479edebe4"},"ver":"1"}
{"reqSignature":{},"txn":{"data":{"data":{"alias":"Node4","blskey":"2zN3bHM1m4rLz54MJHYSwvqzPchYp8jkHswveCLAEJVcX6Mm1wHQD1SkPYMzUDTZvWvhuE6VNAkK3KxVeEmsanSmvjVkReDeBEMxeDaayjcZjFGPydyey1qxBHmTvAnBKoPydvuTAqx5f7YNNRAdeLmUi99gERUU7TD8KfAa6MpQ9bw","blskey_pop":"RPLagxaR5xdimFzwmzYnz4ZhWtYQEj8iR5ZU53T2gitPCyCHQneUn2Huc4oeLd2B2HzkGnjAff4hWTJT6C7qHYB1Mv2wU5iHHGFWkhnTX9WsEAbunJCV2qcaXScKj4tTfvdDKfLiVuU2av6hbsMztirRze7LvYBkRHV3tGwyCptsrP","client_ip":"172.31.35.161","client_port":9708,"node_ip":"172.31.35.161","node_port":9707,"services":["VALIDATOR"]},"dest":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"metadata":{"from":"TWwCRQRZ2ZHMJFn9TzLp7W"},"type":"0"},"txnMetadata":{"seqNo":4,"txnId":"aa5e817d7cc626170eca175822029339a444eb0ee8f0bd20d3b0b76e566fb008"},"ver":"1"}
"indy-node-control.service":
[Unit]
Description=Service for upgrade of existing Indy Node and other operations
#Requires=indy.service
#After=indy.service
After=network.target
[Service]
Type=simple
EnvironmentFile=/etc/indy/node_control.conf
ExecStart=/usr/bin/env python3 -O /usr/local/bin/start_node_control_tool $TEST_MODE --hold-ext ${HOLD_EXT}
Restart=on-failure
RestartSec=10
StartLimitBurst=10
StartLimitInterval=200
TimeoutSec=300
[Install]
WantedBy=multi-user.target
"indy-node.service":
[Unit]
Description=Indy Node
Requires=indy-node-control.service
[Service]
EnvironmentFile=/etc/indy/indy.env
ExecStart=/usr/bin/env python3 -O /usr/local/bin/start_indy_node ${NODE_NAME} ${NODE_IP} ${NODE_PORT} ${NODE_CLIENT_IP} ${NODE_CLIENT_PORT}
User=indy
Group=indy
Restart=on-failure
RestartSec=10
StartLimitBurst=10
StartLimitInterval=200
TimeoutSec=300
LimitNOFILE=65536:131072
[Install]
WantedBy=multi-user.target
"indy.env":
NODE_NAME=Node1
NODE_IP=0.0.0.0
NODE_PORT=9701
NODE_CLIENT_IP=0.0.0.0
NODE_CLIENT_PORT=9702
CLIENT_CONNECTIONS_LIMIT=500
"iptables_config":
"node_control.conf":
# Uncomment this to run agent in test mode:
#TEST_MODE=--test
TEST_MODE=
HOLD_EXT=
In indy_config
enableStdOutLogging = True
LEDGER_DIR = '/var/lib/indy'
LOG_DIR = '/var/log/indy'
KEYS_DIR = '/var/lib/indy'
GENESIS_DIR = '/var/lib/indy'
BACKUP_DIR = '/var/lib/indy/backup'
PLUGINS_DIR = '/var/lib/indy/plugins'
NODE_INFO_DIR = '/var/lib/indy'
indy.env NODE_NAME=Node1 NODE_IP=0.0.0.0 NODE_PORT=9701 NODE_CLIENT_IP=0.0.0.0 NODE_CLIENT_PORT=9702
node_control.conf
TEST_MODE= HOLD_EXT=
I am using Indy Node in Ubuntu 20 outside docker. In docker when I run the load test it doesn't crash as it does in Ubuntu 20. This is the error it gives me
2024-07-05 17:19:08,181|WARNING|stacks.py|Node4C: connections number fell below the limit! Actual: 347, limit: 400. 2024-07-05 17:19:08,181|WARNING|stacks.py|Node4C: client stack restart is not needed anymore. 2024-07-05 17:19:08,878|WARNING|stacks.py|Node4C: connections limit reached! Actual: 410, limit: 400. 2024-07-05 17:19:08,878|WARNING|stacks.py|Node4C: connections limit reached but too few time spent since client stack start, restart it later. 2024-07-05 17:19:15,063|INFO|ordering_service.py|Node4:1 set last ordered as (0, 21) 2024-07-05 17:19:15,071|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 21, ledger 1, state root None, txn root None, audit root None, requests ordered 532, discarded 0 2024-07-05 17:19:15,075|INFO|ordering_service.py|Node4:1 set last ordered as (0, 22) 2024-07-05 17:19:15,081|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 22, ledger 1, state root None, txn root None, audit root None, requests ordered 182, discarded 0 2024-07-05 17:19:15,081|INFO|ordering_service.py|Node4:1 set last ordered as (0, 23) 2024-07-05 17:19:15,086|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 23, ledger 1, state root None, txn root None, audit root None, requests ordered 473, discarded 0 2024-07-05 17:19:15,768|INFO|ordering_service.py|Node4:1 set last ordered as (0, 24) 2024-07-05 17:19:15,769|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 24, ledger 1, state root None, txn root None, audit root None, requests ordered 37, discarded 0 2024-07-05 17:19:24,202|INFO|ordering_service.py|Node4:0 set last ordered as (0, 16) 2024-07-05 17:19:25,005|INFO|ordering_service.py|Node4:0 ordered batch request, view no 0, ppSeqNo 16, ledger 1, state root J3kBWhVEkrhbtwhw7tuB1Y78o49JMSfoUebFvX96xAcc, txn root B9TH7NwF4M6fw6Wk6qYJRBX1VTJjnvKJZPfTSCFGiGPU, audit root 56dnDyFMkxPDTr5xNpTvzfbYFPhhdkDePSGjz5TBuxer, requests ordered 1000, discarded 0 2024-07-05 17:19:33,007|NOTIFICATION|looper.py|Looper shutting down now... 2024-07-05 17:19:33,007|INFO|motor.py|Node4 changing status from started to stopping 2024-07-05 17:19:33,007|INFO|node.py|node Node4 current stats
node inbox size : 0 client inbox size : 0 age (seconds) : 155.4772744178772 next check for reconnect: 6.888445964999988 node connections : {'Node3', 'Node2', 'Node1'} f : 1 master instance : 0 replicas : 2 view no : 0 rank : 3 msgs to replicas : 7 action queue : 0 140633492501984 action queue stash : 4 140633492502080 2024-07-05 17:19:33,008|INFO|node.py|Node4 reseting... 2024-07-05 17:19:33,011|NOTIFICATION|zstack.py|stack Node4 closing its listener 2024-07-05 17:19:33,012|NOTIFICATION|zstack.py|stack Node4 stopped 2024-07-05 17:19:33,012|NOTIFICATION|zstack.py|stack Node4C closing its listener
And all the nodes crash. My specs are quite good in each of the ubuntu instance. 16GB RAM and 8 CPU. The dockerized version of indy node is half of it and doesn't crash like this. What should I do? Thanks