search

hyperledger / indy-node

The server portion of a distributed ledger purpose-built for decentralized identity.
https://wiki.hyperledger.org/display/indy
Apache License 2.0
685 stars 657 forks source link

Indy Node crashes and closes its listener on heavy load or load for longer period #1868

Open SuryaSHalwasia opened 4 months ago

SuryaSHalwasia commented 4 months ago

I am using Indy Node in Ubuntu 20 outside docker. In docker when I run the load test it doesn't crash as it does in Ubuntu 20. This is the error it gives me

2024-07-05 17:19:08,181|WARNING|stacks.py|Node4C: connections number fell below the limit! Actual: 347, limit: 400. 2024-07-05 17:19:08,181|WARNING|stacks.py|Node4C: client stack restart is not needed anymore. 2024-07-05 17:19:08,878|WARNING|stacks.py|Node4C: connections limit reached! Actual: 410, limit: 400. 2024-07-05 17:19:08,878|WARNING|stacks.py|Node4C: connections limit reached but too few time spent since client stack start, restart it later. 2024-07-05 17:19:15,063|INFO|ordering_service.py|Node4:1 set last ordered as (0, 21) 2024-07-05 17:19:15,071|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 21, ledger 1, state root None, txn root None, audit root None, requests ordered 532, discarded 0 2024-07-05 17:19:15,075|INFO|ordering_service.py|Node4:1 set last ordered as (0, 22) 2024-07-05 17:19:15,081|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 22, ledger 1, state root None, txn root None, audit root None, requests ordered 182, discarded 0 2024-07-05 17:19:15,081|INFO|ordering_service.py|Node4:1 set last ordered as (0, 23) 2024-07-05 17:19:15,086|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 23, ledger 1, state root None, txn root None, audit root None, requests ordered 473, discarded 0 2024-07-05 17:19:15,768|INFO|ordering_service.py|Node4:1 set last ordered as (0, 24) 2024-07-05 17:19:15,769|INFO|ordering_service.py|Node4:1 ordered batch request, view no 0, ppSeqNo 24, ledger 1, state root None, txn root None, audit root None, requests ordered 37, discarded 0 2024-07-05 17:19:24,202|INFO|ordering_service.py|Node4:0 set last ordered as (0, 16) 2024-07-05 17:19:25,005|INFO|ordering_service.py|Node4:0 ordered batch request, view no 0, ppSeqNo 16, ledger 1, state root J3kBWhVEkrhbtwhw7tuB1Y78o49JMSfoUebFvX96xAcc, txn root B9TH7NwF4M6fw6Wk6qYJRBX1VTJjnvKJZPfTSCFGiGPU, audit root 56dnDyFMkxPDTr5xNpTvzfbYFPhhdkDePSGjz5TBuxer, requests ordered 1000, discarded 0 2024-07-05 17:19:33,007|NOTIFICATION|looper.py|Looper shutting down now... 2024-07-05 17:19:33,007|INFO|motor.py|Node4 changing status from started to stopping 2024-07-05 17:19:33,007|INFO|node.py|node Node4 current stats

node inbox size : 0 client inbox size : 0 age (seconds) : 155.4772744178772 next check for reconnect: 6.888445964999988 node connections : {'Node3', 'Node2', 'Node1'} f : 1 master instance : 0 replicas : 2 view no : 0 rank : 3 msgs to replicas : 7 action queue : 0 140633492501984 action queue stash : 4 140633492502080 2024-07-05 17:19:33,008|INFO|node.py|Node4 reseting... 2024-07-05 17:19:33,011|NOTIFICATION|zstack.py|stack Node4 closing its listener 2024-07-05 17:19:33,012|NOTIFICATION|zstack.py|stack Node4 stopped 2024-07-05 17:19:33,012|NOTIFICATION|zstack.py|stack Node4C closing its listener

And all the nodes crash. My specs are quite good in each of the ubuntu instance. 16GB RAM and 8 CPU. The dockerized version of indy node is half of it and doesn't crash like this. What should I do? Thanks

WadeBarnes commented 4 months ago

It doesn't look like the node(s) are crashing. It looks more like the indy-node-control service is restarting them (possibly due to performance while under load). The indy-node-control service does not run in the docker containers.

Could you please provide the following information (redact any private info):

/var/log/indy/<network_name>/<node_alias>.log
/var/log/indy/node_control.log
sudo journalctl -u indy-node
sudo journalctl -u indy-node-control
sudo validator-info -v
cat /etc/indy/indy_config.py
cat /etc/indy/indy.env
cat /etc/indy/node_control.conf
SuryaSHalwasia commented 4 months ago

So here are the details In /var/log/indy/sandbox/Node1.log 2024-07-07 23:36:28,927|INFO|ordering_service.py|Node1:0 ordered batch request, view no 0, ppSeqNo 11, ledger 1, state root DfuSUSvHxfsAmBHE1sj9anpaa8vsMDfRx9KbxqMSneFS, txn root BVnQVtMdG7chJR8M2Ge6s4GRaoPXkzHpbEyTDeLw2ypB, audit root 5R9kkYBz3xJDrAet4DYchGWpmxWNdJwYEN1nG8smQX9V, requests ordered 783, discarded 0 2024-07-07 23:36:35,646|NOTIFICATION|looper.py|Looper shutting down now... 2024-07-07 23:36:35,646|INFO|motor.py|Node1 changing status from started to stopping 2024-07-07 23:36:35,646|INFO|node.py|node Node1 current stats

node inbox size : 0 client inbox size : 0 age (seconds) : 352.6427562236786 next check for reconnect: 2.526324950000344 node connections : {'Node2', 'Node3', 'Node4'} f : 1 master instance : 0 replicas : 2 view no : 0 rank : 0 msgs to replicas : 0 action queue : 0 139717908141536 action queue stash : 5 139717908141632 2024-07-07 23:36:35,646|INFO|node.py|Node1 reseting... 2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1 closing its listener 2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1 stopped 2024-07-07 23:36:35,660|NOTIFICATION|zstack.py|stack Node1C closing its listener 2024-07-07 23:36:35,664|NOTIFICATION|zstack.py|stack Node1C stopped 2024-07-07 23:36:35,665|INFO|node.py|Node1 closing key-value storages 2024-07-07 23:36:36,658|INFO|motor.py|Node1 changing status from stopping to stopped 2024-07-07 23:36:36,658|NOTIFICATION|looper.py|Looper shut down in 1.012 seconds.

In others it is 2024-07-07 23:36:35,767|NOTIFICATION|keep_in_touch.py|Node3's connections changed from {'Node2', 'Node4', 'Node1'} to {'Node2', 'Node4'} 2024-07-07 23:36:35,767|NOTIFICATION|keep_in_touch.py|CONNECTION: Node3 disconnected from Node1

2024-07-07 23:36:35,767|INFO|motor.py|Node3 changing status from started to started_hungry 2024-07-07 23:36:35,767|NOTIFICATION|primary_connection_monitor_service.py|Node3:0 lost connection to primary of master 2024-07-07 23:36:35,767|INFO|primary_connection_monitor_service.py|Node3:0 scheduling primary connection check in 60 sec

In /var/log/indy/node_control.log (This was only present in Node1) 2024-06-26 20:15:24,574|INFO|node_control_tool.py|Node control tool is starting up on localhost port 30003 2024-06-26 20:15:25,142|INFO|node_control_utils.py|Successfully put indy-plenum indy-node python3-ioflo python3-orderedset python3-base58 python3-prompt-toolkit python3-rlp python3-sha3 python3-libnacl python3-six python3-portalocker python3-sortedcontainers python3-dateutil python3-semver python3-psutil python3-pyzmq python3-jsonpickle python3-rocksdb python3-pympler python3-packaging python3-timeout-decorator python3-distro packages on hold 2024-06-26 20:15:25,142|DEBUG|node_control_tool.py|Waiting for the next event

For journalctl -u indy-node -- Logs begin at Sat 2023-12-02 18:58:18 UTC, end at Mon 2024-07-08 14:56:20 UTC. -- -- No entries --

For indy-node-control

-- Logs begin at Sat 2023-12-02 18:58:18 UTC, end at Mon 2024-07-08 14:58:55 UTC. -- Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Started Service for upgrade of existing Indy Node and other operations. Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: indy-node-control.service: Succeeded. Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations. Jun 26 20:15:23 ip-172-31-40-65 systemd[1]: Started Service for upgrade of existing Indy Node and other operations. Jun 26 20:15:24 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:24,574|INFO|node_control_tool.py|Node control tool is starting up o> Jun 26 20:15:25 ip-172-31-40-65 env[19732]: indy-plenum set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: indy-node set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-ioflo set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-orderedset set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-base58 set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-prompt-toolkit set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-rlp set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-sha3 set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-libnacl set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-six set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-portalocker set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-sortedcontainers set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-dateutil set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-semver set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-psutil set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-pyzmq set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-jsonpickle set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-rocksdb set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-pympler set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-packaging set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-timeout-decorator set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19732]: python3-distro set on hold. Jun 26 20:15:25 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:25,142|INFO|node_control_utils.py|Successfully put indy-plenum indy> Jun 26 20:15:25 ip-172-31-40-65 env[19720]: 2024-06-26 20:15:25,142|DEBUG|node_control_tool.py|Waiting for the next event Jun 27 04:22:39 ip-172-31-40-65 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Jun 27 04:22:40 ip-172-31-40-65 systemd[1]: indy-node-control.service: Succeeded. Jun 27 04:22:40 ip-172-31-40-65 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations.

For validator-info "Hardware": "HDD_all": 6405 Mbs "RAM_all_free": 31119 Mbs "RAM_used_by_node": 687 Mbs "Node_info": "BLS_key": 4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba "Catchup_status": "Last_txn_3PC_keys": "0": "Node2": None None "Node3": None None "1": "Node2": None None "Node3": None None "2": "Node2": None None "Node3": None None "3": "Node2": None None "Node3": None None "Ledger_statuses": "0": synced "1": synced "2": synced "3": synced "Number_txns_in_catchup": "0": 0 "1": 0 "2": 0 "3": 0 "Received_LedgerStatus": "Waiting_consistency_proof_msgs": "0": None "1": None "2": None "3": None "Client_ip": 0.0.0.0 "Client_port": 9702 "Client_protocol": tcp "Committed_ledger_root_hashes": "0": b'2PZguAeNfTpfPsavVTfAq98UC89Ko4DgbBa2CpTL1Svv' "1": b'HgxLc6h6R12P5zSM1h7T5jSttmAoHryhmrshffmM9imx' "2": b'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn' "3": b'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn' "Committed_state_root_hashes": "0": b'EQUSi2crBb6G82h8NdUjU8MBNQaHTxRtwvtu9Qi1PAG' "1": b'39aVhSB3idDKLkKTUGfKHFMhE2oNZ5GyCSDqCg4M5x2U' "2": b'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA' "Count_of_replicas": 2 "Freshness_status": "0": "Has_write_consensus": True "Last_updated_time": 2024-07-07 23:30:43+00:00 "1": "Has_write_consensus": True "Last_updated_time": 2024-07-07 23:30:43+00:00 "2": "Has_write_consensus": True "Last_updated_time": 2024-07-07 23:30:43+00:00 "Metrics": "Delta": 0.1 "Lambda": 240 "Omega": 20 "average-per-second": "read-transactions": 0.0 "write-transactions": 0.0 "avg backup throughput": 0.0 "client avg request latencies": "0": None "1": None "instances started": "0": 1785.408685558 "1": 1785.410516859 "master throughput": None "master throughput ratio": None "max master request latencies": 0 "ordered request counts": "0": 0 "1": 0 "ordered request durations": "0": 0 "1": 0 "throughput": "0": 0.0 "1": 0.0 "total requests": 0 "transaction-count": "audit": 0 "config": 0 "ledger": 10 "pool": 4 "uptime": 300 "Mode": participating "Name": Node1 "Node_ip": 0.0.0.0 "Node_port": 9701 "Node_protocol": tcp "Replicas_status": "Node1:0": "Last_ordered_3PC": 0 0 "Primary": Node1:0 "Stashed_txns": "Stashed_PrePrepare": 0 "Stashed_checkpoints": 0 "Watermarks": 0:300 "Node1:1": "Last_ordered_3PC": 0 0 "Primary": Node2:1 "Stashed_txns": "Stashed_PrePrepare": 0 "Stashed_checkpoints": 0 "Watermarks": 0:9223372036854775807 "Requests_timeouts": "Ordering_phase_req_timeouts": 0 "Propagates_phase_req_timeouts": 0 "Uncommitted_ledger_root_hashes": "Uncommitted_ledger_txns": "0": "Count": 0 "1": "Count": 0 "2": "Count": 0 "3": "Count": 0 "Uncommitted_state_root_hashes": "0": b'EQUSi2crBb6G82h8NdUjU8MBNQaHTxRtwvtu9Qi1PAG' "1": b'39aVhSB3idDKLkKTUGfKHFMhE2oNZ5GyCSDqCg4M5x2U' "2": b'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA' "View_change_status": "IC_queue": "1": "Voters": "Node2": "reason": 26 "Node3": "reason": 26 "Last_complete_view_no": 0 "Last_view_change_started_at": 1970-01-01 00:00:00 "VCDone_queue": "VC_in_progress": False "View_No": 0 "did": Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv "verkey": 33nHHYKnqmtGAVfZZGoP8hpeExeH45Fo8cKmd5mcnKYk7XgWNBxkkKJ "Pool_info": "Blacklisted_nodes": "Quorums": {'n': 4, 'f': 1, 'weak': Quorum(2), 'strong': Quorum(3), 'propagate': Quorum(2), 'prepare': Quorum(2), 'commit': Quorum(3), 'reply': Quorum(2), 'view_change': Quorum(3), 'election': Quorum(3), 'view_change_ack': Quorum(2), 'view_change_done': Quorum(3), 'same_consistency_proof': Quorum(2), 'consistency_proof': Quorum(2), 'ledger_status': Quorum(2), 'ledger_status_last_3PC': Quorum(2), 'checkpoint': Quorum(2), 'timestamp': Quorum(2), 'bls_signatures': Quorum(3), 'observer_data': Quorum(2), 'backup_instance_faulty': Quorum(2)} "Reachable_nodes": ['Node1', 0] ['Node2', 1] ['Node3', None] ['Node4', None] "Reachable_nodes_count": 4 "Read_only": False "Suspicious_nodes": "Total_nodes_count": 4 "Unreachable_nodes": "Unreachable_nodes_count": 0 "f_value": 1 "Protocol": "Software": "Indy_packages": hi indy-node 1.13.2~rc6 amd64 Indy node hi indy-plenum 1.13.1~rc4 amd64 Plenum Byzantine Fault Tolerant Protocol

 "Installed_packages":
       ioflo 2.0.2
       tzdata 2024.1
       distro 1.7.0
       indy-plenum 1.13.1rc4
       pillow 10.3.0
       rlp 0.6.0
       fonttools 4.53.0
       ujson 1.33
       python-rocksdb 0.7.0
       importlib_resources 6.4.0
       jsonpickle 3.0.1
       kiwisolver 1.4.5
       importlib-metadata 3.10.1
       zipp 3.19.2
       cycler 0.12.1
       python-ursa 0.1.1
       pyzmq 22.3.0
       leveldb 0.201
       matplotlib 3.7.5
       Pympler 0.8
       python-dateutil 2.8.2
       setuptools 50.3.2
       msgpack-python 0.5.6
       packaging 21.3
       timeout-decorator 0.5.0
       psutil 5.9.4
       pandas 2.0.3
       semver 2.13.0
       base58 2.1.1
       indy-node 1.13.2rc6
       orderedset 2.0.3
       sha3 0.2.1
       portalocker 2.7.0
       libnacl 1.6.1
       sortedcontainers 1.5.7
       contourpy 1.1.1
       prompt-toolkit 3.0.18
       numpy 1.24.4
       pytz 2024.1
       urllib3 1.25.8
       unattended-upgrades 0.1
       language-selector 0.1
       pyrsistent 0.15.5
       ubuntu-pro-client 8001
       SecretStorage 2.3.1
       pyOpenSSL 19.0.0
       zipp 1.0.0
       netifaces 0.10.4
       systemd-python 234
       PyJWT 1.7.1
       Automat 0.8.0
       requests 2.22.0
       sos 4.5.6
       PyYAML 5.3.1
       keyring 18.0.1
       more-itertools 4.2.0
       command-not-found 0.3
       hibagent 1.0.1
       jsonpointer 2.0
       pyasn1 0.4.2
       pluggy 0.13.0
       cryptography 2.8
       ec2-hibinit-agent 1.0.0
       idna 2.8
       PyHamcrest 1.9.0
       Twisted 18.9.0
       six 1.14.0
       python-apt 2.0.1+ubuntu0.20.4.1
       MarkupSafe 1.1.0
       python-debian 0.1.36+ubuntu1.1
       wadllib 1.3.3
       blinker 1.4
       hyperlink 19.0.0
       zope.interface 4.7.1
       pyasn1-modules 0.2.1
       PyNaCl 1.3.0
       pexpect 4.6.0
       Click 7.0
       httplib2 0.14.0
       certifi 2019.11.28
       jsonpatch 1.22
       pyparsing 2.4.6
       pyserial 3.4
       dbus-python 1.2.16
       service-identity 18.1.0
       entrypoints 0.3
       colorama 0.4.3
       oauthlib 3.1.0
       chardet 3.0.4
       PyGObject 3.36.0
       cloud-init 24.1.3
       distro-info 0.23+ubuntu1.1
       lazr.uri 1.0.3
       constantly 15.1.0
       attrs 19.3.0
       pip 20.0.2
       setuptools 45.2.0
       ufw 0.36
       wheel 0.34.2
       jsonschema 3.2.0
       Jinja2 2.10.1
       ssh-import-id 5.10
       requests-unixsocket 0.2.0
       launchpadlib 1.10.13
       incremental 16.10.1
       simplejson 3.16.0
       pymacaroons 0.13.0
       pytest 4.6.9
       atomicwrites 1.1.5
       wcwidth 0.1.8
       lazr.restfulclient 0.14.2
       configobj 5.0.6
       py 1.8.1
 "OS_version":   Linux-5.15.0-1064-aws-x86_64-with-glibc2.29
 "indy-node":    1.13.2rc6
 "sovrin":       unknown

"Update_time": Sunday, July 7, 2024 11:35:43 PM +0000 "response-version": 0.0.1 "timestamp": 1720395343

"Configuration": "Config": "Main_config":

Current network

            # Disable stdout logging
            enableStdOutLogging = True
            #logLevel=0
            # Directory to store ledger.
            LEDGER_DIR = '/var/lib/indy'

            # Directory to store logs.
            LOG_DIR = '/var/log/indy'

            # Directory to store keys.
            KEYS_DIR = '/var/lib/indy'

            # Directory to store genesis transactions files.
            GENESIS_DIR = '/var/lib/indy'

            # Directory to store backups.
            BACKUP_DIR = '/var/lib/indy/backup'

            # Directory to store plugins.
            PLUGINS_DIR = '/var/lib/indy/plugins'

            # Directory to store node info.
            NODE_INFO_DIR = '/var/lib/indy'
            NETWORK_NAME = 'sandbox'
      "Network_config":
      "User_config":
 "Genesis_txns":
      "domain_txns":
            {"reqSignature":{},"txn":{"data":{"dest":"V4SGRU86Z58d6TV7PBUe6f","role":"0","verkey":"~CoRER63DVYnWZtK8uAzNbx"},"metadata":{},"type":"1"},"txnMetadata":{"seqNo":1},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"Th7MpTaRZVRYnPiabds81Y","role":"2","verkey":"~7TYfekw4GUagBnBVCqPjiC"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":2},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"EbP4aYNeTHL6q385GuVpRV","role":"2","verkey":"~RHGNtfvkgPEUQzQNtNxLNu"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":3},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"4cU41vWW82ArfxJxHkzXPG","role":"2","verkey":"~EMoPA6HrpiExVihsVfxD3H"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":4},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"TWwCRQRZ2ZHMJFn9TzLp7W","role":"2","verkey":"~UhP7K35SAXbix1kCQV4Upx"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":5},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"7JhapNNMLnwkbiC2ZmPZSE","verkey":"~LgpYPrzkB6awcHMTPZ9TVn"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":6},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"MEPecrczs4Wh6FA12u519D","verkey":"~A4rMgHYboWYS1DXibCgo9W"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":7},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"EAPtwgevBpzP8hkj9sxuzy","verkey":"~gmzSzu3feXC6g2djF7ar4"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":8},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"LuL1HK1sDruwkfm68jrVfD","verkey":"~Nyv9BKUJuvjgMbfbwk8CFD"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":9},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"dest":"462p8mtcX6jpa9ky565YEL","verkey":"~LCgq4hnSvMvB8nKd9vgsTD"},"metadata":{"from":"V4SGRU86Z58d6TV7PBUe6f"},"type":"1"},"txnMetadata":{"seqNo":10},"ver":"1"}

      "pool_txns":
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node1","blskey":"4N8aUNHSgjQVgkpm8nhNEfDf6txHznoYREg9kirmJrkivgL4oSEimFF6nsQ6M41QvhM2Z33nves5vfSn9n1UwNFJBYtWVnHYMATn76vLuL3zU88KyeAYcHfsih3He6UHcXDxcaecHVz6jhCYz1P2UZn2bDVruL5wXpehgBfBaLKm3Ba","blskey_pop":"RahHYiCvoNCtPTrVtP7nMC5eTYrsUA8WjXbdhNc8debh1agE9bGiJxWBXYNFbnJXoXhWFMvyqhqhRoq737YQemH5ik9oL7R4NTTCz2LEZhkgLJzB3QRQqJyBNyv7acbdHrAT8nQ9UkLbaVL9NBpnWXBTw4LEMePaSHEw66RzPNdAX1","client_ip":"172.31.40.65","client_port":9702,"node_ip":"172.31.40.65","node_port":9701,"services":["VALIDATOR"]},"dest":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"metadata":{"from":"Th7MpTaRZVRYnPiabds81Y"},"type":"0"},"txnMetadata":{"seqNo":1,"txnId":"fea82e10e894419fe2bea7d96296a6d46f50f93f9eeda954ec461b2ed2950b62"},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node2","blskey":"37rAPpXVoxzKhz7d9gkUe52XuXryuLXoM6P6LbWDB7LSbG62Lsb33sfG7zqS8TK1MXwuCHj1FKNzVpsnafmqLG1vXN88rt38mNFs9TENzm4QHdBzsvCuoBnPH7rpYYDo9DZNJePaDvRvqJKByCabubJz3XXKbEeshzpz4Ma5QYpJqjk","blskey_pop":"Qr658mWZ2YC8JXGXwMDQTzuZCWF7NK9EwxphGmcBvCh6ybUuLxbG65nsX4JvD4SPNtkJ2w9ug1yLTj6fgmuDg41TgECXjLCij3RMsV8CwewBVgVN67wsA45DFWvqvLtu4rjNnE9JbdFTc1Z4WCPA3Xan44K1HoHAq9EVeaRYs8zoF5","client_ip":"172.31.40.114","client_port":9704,"node_ip":"172.31.40.114","node_port":9703,"services":["VALIDATOR"]},"dest":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"metadata":{"from":"EbP4aYNeTHL6q385GuVpRV"},"type":"0"},"txnMetadata":{"seqNo":2,"txnId":"1ac8aece2a18ced660fef8694b61aac3af08ba875ce3026a160acbc3a3af35fc"},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node3","blskey":"3WFpdbg7C5cnLYZwFZevJqhubkFALBfCBBok15GdrKMUhUjGsk3jV6QKj6MZgEubF7oqCafxNdkm7eswgA4sdKTRc82tLGzZBd6vNqU8dupzup6uYUf32KTHTPQbuUM8Yk4QFXjEf2Usu2TJcNkdgpyeUSX42u5LqdDDpNSWUK5deC5","blskey_pop":"QwDeb2CkNSx6r8QC8vGQK3GRv7Yndn84TGNijX8YXHPiagXajyfTjoR87rXUu4G4QLk2cF8NNyqWiYMus1623dELWwx57rLCFqGh7N4ZRbGDRP4fnVcaKg1BcUxQ866Ven4gw8y4N56S5HzxXNBZtLYmhGHvDtk6PFkFwCvxYrNYjh","client_ip":"172.31.32.116","client_port":9706,"node_ip":"172.31.32.116","node_port":9705,"services":["VALIDATOR"]},"dest":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"metadata":{"from":"4cU41vWW82ArfxJxHkzXPG"},"type":"0"},"txnMetadata":{"seqNo":3,"txnId":"7e9f355dffa78ed24668f0e0e369fd8c224076571c51e2ea8be5f26479edebe4"},"ver":"1"}
            {"reqSignature":{},"txn":{"data":{"data":{"alias":"Node4","blskey":"2zN3bHM1m4rLz54MJHYSwvqzPchYp8jkHswveCLAEJVcX6Mm1wHQD1SkPYMzUDTZvWvhuE6VNAkK3KxVeEmsanSmvjVkReDeBEMxeDaayjcZjFGPydyey1qxBHmTvAnBKoPydvuTAqx5f7YNNRAdeLmUi99gERUU7TD8KfAa6MpQ9bw","blskey_pop":"RPLagxaR5xdimFzwmzYnz4ZhWtYQEj8iR5ZU53T2gitPCyCHQneUn2Huc4oeLd2B2HzkGnjAff4hWTJT6C7qHYB1Mv2wU5iHHGFWkhnTX9WsEAbunJCV2qcaXScKj4tTfvdDKfLiVuU2av6hbsMztirRze7LvYBkRHV3tGwyCptsrP","client_ip":"172.31.35.161","client_port":9708,"node_ip":"172.31.35.161","node_port":9707,"services":["VALIDATOR"]},"dest":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"metadata":{"from":"TWwCRQRZ2ZHMJFn9TzLp7W"},"type":"0"},"txnMetadata":{"seqNo":4,"txnId":"aa5e817d7cc626170eca175822029339a444eb0ee8f0bd20d3b0b76e566fb008"},"ver":"1"}

 "indy-node-control.service":
       [Unit]
       Description=Service for upgrade of existing Indy Node and other operations
       #Requires=indy.service
       #After=indy.service
       After=network.target

       [Service]
       Type=simple
       EnvironmentFile=/etc/indy/node_control.conf
       ExecStart=/usr/bin/env python3 -O /usr/local/bin/start_node_control_tool $TEST_MODE --hold-ext ${HOLD_EXT}
       Restart=on-failure
       RestartSec=10
       StartLimitBurst=10
       StartLimitInterval=200
       TimeoutSec=300

       [Install]
       WantedBy=multi-user.target

 "indy-node.service":
       [Unit]
       Description=Indy Node
       Requires=indy-node-control.service

       [Service]
       EnvironmentFile=/etc/indy/indy.env
       ExecStart=/usr/bin/env python3 -O /usr/local/bin/start_indy_node ${NODE_NAME} ${NODE_IP} ${NODE_PORT} ${NODE_CLIENT_IP} ${NODE_CLIENT_PORT}
       User=indy
       Group=indy
       Restart=on-failure
       RestartSec=10
       StartLimitBurst=10
       StartLimitInterval=200
       TimeoutSec=300
       LimitNOFILE=65536:131072

       [Install]
       WantedBy=multi-user.target

 "indy.env":
       NODE_NAME=Node1
       NODE_IP=0.0.0.0
       NODE_PORT=9701
       NODE_CLIENT_IP=0.0.0.0
       NODE_CLIENT_PORT=9702
       CLIENT_CONNECTIONS_LIMIT=500
 "iptables_config":
 "node_control.conf":
       # Uncomment this to run agent in test mode:
       #TEST_MODE=--test

       TEST_MODE=
       HOLD_EXT=

In indy_config

Current network

Disable stdout logging

enableStdOutLogging = True

Directory to store ledger.

LEDGER_DIR = '/var/lib/indy'

Directory to store logs.

LOG_DIR = '/var/log/indy'

Directory to store keys.

KEYS_DIR = '/var/lib/indy'

Directory to store genesis transactions files.

GENESIS_DIR = '/var/lib/indy'

Directory to store backups.

BACKUP_DIR = '/var/lib/indy/backup'

Directory to store plugins.

PLUGINS_DIR = '/var/lib/indy/plugins'

Directory to store node info.

NODE_INFO_DIR = '/var/lib/indy'

indy.env NODE_NAME=Node1 NODE_IP=0.0.0.0 NODE_PORT=9701 NODE_CLIENT_IP=0.0.0.0 NODE_CLIENT_PORT=9702

node_control.conf

Uncomment this to run agent in test mode:

TEST_MODE=--test

TEST_MODE= HOLD_EXT=