Running issues: von-network nodes fail to start

[Qingbao]

Could not start the nodes and web, tried clean and restart

$ ./manage start ./manage: line 6: ifconfig: command not found ./manage: line 9: ifconfig: command not found The IP variable is not set. Defaulting to a blank string. The IPS variable is not set. Defaulting to a blank string. Creating volume "von_node1-data" with default driver Creating volume "von_node2-data" with default driver Creating volume "von_node3-data" with default driver Creating volume "von_node4-data" with default driver Creating volume "von_von-client-cli" with default driver Creating volume "von_von-client-libindy" with default driver Creating volume "von_von-web-indy" with default driver Creating von_node4_1 ... done Creating von_node2_1 ... done Creating von_node3_1 ... done Creating von_node1_1 ... done Creating von_von-web_1 ... done Attaching to von_node2_1, von_node4_1, von_node3_1, von_node1_1, von_von-web_1 node2_1 | Ledger does not exist - recreating... node2_1 | von_generate_transactions -n 2 node2_1 | init_indy_node Node2 9418 80 node2_1 | Node-stack name is Node2 node2_1 | Client-stack name is Node2C node2_1 | Generating keys for random seed b'9C3e487AecB3087B7adDDe64b93a1F7E' node2_1 | Init local keys for client-stack node2_1 | Public key is fbd58bf8694cae02ebe4fdaf36b68794de6081ca25938e075a2970d5ae2a254e node2_1 | Verification key is cd7eeab8b3bdd8efa037392d3b8c676c48c1b396dc64d735fdf9cd316e804e4c node2_1 | Init local keys for node-stack node2_1 | Public key is fbd58bf8694cae02ebe4fdaf36b68794de6081ca25938e075a2970d5ae2a254e node2_1 | Verification key is cd7eeab8b3bdd8efa037392d3b8c676c48c1b396dc64d735fdf9cd316e804e4c node2_1 | BLS Public key is 3b59vKQwLSD8E7EKaburnCkPeffNfNSCyG6XPv1vnKhQJ9qgjWNhhbqtqHSKdvZ9MzSR3cbw9y2Dxp4SkFDyJks6hZXPLm6z8S7EgHUX8aAQE2B2r18C7VrgepWA2GmP8KbyMYogS78sqMeWnEZo84nzmKhBrPWeKXCj3WDWx6S5B6v node2_1 | generate_indy_pool_transactions --nodes 4 --clients 0 --nodeNum 2 --ips ,,, node2_1 | Traceback (most recent call last): node2_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 253, in _bootstrap_args_type_ips_hosts node2_1 | ipaddress.ip_address(arg) node2_1 | File "/usr/lib/python3.5/ipaddress.py", line 54, in ip_address node2_1 | address) node2_1 | ValueError: '' does not appear to be an IPv4 or IPv6 address node2_1 | node2_1 | During handling of the above exception, another exception occurred: node2_1 | node2_1 | Traceback (most recent call last): node2_1 | File "/usr/local/bin/generate_indy_pool_transactions", line 19, in node2_1 | getTxnOrderedFields(), ConfigHelper, NodeConfigHelper) node2_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 197, in bootstrapTestNodes node2_1 | args = parser.parse_args() node2_1 | File "/usr/lib/python3.5/argparse.py", line 1735, in parse_args node2_1 | args, argv = self.parse_known_args(args, namespace) node2_1 | File "/usr/lib/python3.5/argparse.py", line 1767, in parse_known_args node2_1 | namespace, args = self._parse_known_args(args, namespace) node2_1 | File "/usr/lib/python3.5/argparse.py", line 1973, in _parse_known_args node2_1 | start_index = consume_optional(start_index) node2_1 | File "/usr/lib/python3.5/argparse.py", line 1913, in consume_optional node2_1 | take_action(action, args, option_string) node2_1 | File "/usr/lib/python3.5/argparse.py", line 1825, in take_action node2_1 | argument_values = self._get_values(action, argument_strings) node2_1 | File "/usr/lib/python3.5/argparse.py", line 2266, in _get_values node2_1 | value = self._get_value(action, arg_string) node2_1 | File "/usr/lib/python3.5/argparse.py", line 2295, in _get_value node2_1 | result = type_func(arg_string) node2_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 255, in _bootstrap_args_type_ips_hosts node2_1 | if not is_hostname_valid(arg): node2_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/util.py", line 538, in is_hostname_valid node2_1 | if hostname[-1] == ".": node2_1 | IndexError: string index out of range node4_1 | Ledger does not exist - recreating... node4_1 | von_generate_transactions -n 4 node4_1 | init_indy_node Node4 9418 80 node4_1 | Node-stack name is Node4 node4_1 | Client-stack name is Node4C node4_1 | Generating keys for random seed b'aE573Ecb10EdecDAA3fcF66FEDEbC6Ce' node4_1 | Init local keys for client-stack node4_1 | Public key is 45b7e3082505c69b6f714fe5e94baccf53cc106a595800c3d983da20ba9ade65 node4_1 | Verification key is 51f9c4b596d831c4b7ae8b45bfa46031ee46540737df9fb6503bcb27784511ff node4_1 | Init local keys for node-stack node4_1 | Public key is 45b7e3082505c69b6f714fe5e94baccf53cc106a595800c3d983da20ba9ade65 node4_1 | Verification key is 51f9c4b596d831c4b7ae8b45bfa46031ee46540737df9fb6503bcb27784511ff node4_1 | BLS Public key is Kh2ptKhNxoWJNXkdCCBbfWfK4qv8EskEzuY3sFqR6QsuB8pXT5UJYsRcFoMcZceuh8Rs4Q6ofiRqFU5XJ9wz5hFHStncyp4SsnCrKXKbAfmQqq1TYJi2sguCYAQSLJH3hBtLwAf4YX3BbX8wKcEjNKfsgeZUx6QXGNRnJMq3h5H4Rs node4_1 | generate_indy_pool_transactions --nodes 4 --clients 0 --nodeNum 4 --ips ,,, node3_1 | Ledger does not exist - recreating... node3_1 | von_generate_transactions -n 3 node3_1 | init_indy_node Node3 9418 80 node3_1 | Node-stack name is Node3 node3_1 | Client-stack name is Node3C node3_1 | Generating keys for random seed b'c95Bf30ccac53f9DfAa26279C613ec94' node3_1 | Init local keys for client-stack node3_1 | Public key is bba4d2a8214311b769c1ed23eaaf98132d977a0b526e178a3303f5f3335f6849 node3_1 | Verification key is c67c2642cfec4121572300579baafa41d04b224fe735f7d5ceaafc5b2d81ef19 node3_1 | Init local keys for node-stack node3_1 | Public key is bba4d2a8214311b769c1ed23eaaf98132d977a0b526e178a3303f5f3335f6849 node3_1 | Verification key is c67c2642cfec4121572300579baafa41d04b224fe735f7d5ceaafc5b2d81ef19 node3_1 | BLS Public key is 32y8CKvvnfCzgD7Umk4h63KC1Kk4Uvaj4Nb8QScEebtvXZtFhhYYNRFAWsBeK4Fhbf48spJvnKgHukwjHeqoBJnQ8o4hZtv1zeo3h3yZQxQD6XbHh1eY5wrG3HgHCFcZn12kzsPnJj6NSy5mgvYk5yL2PNznc16xjKhaMWJj8D6yf6r node3_1 | generate_indy_pool_transactions --nodes 4 --clients 0 --nodeNum 3 --ips ,,, node1_1 | Ledger does not exist - recreating... node1_1 | von_generate_transactions -n 1 node1_1 | init_indy_node Node1 9418 80 von-web_1 | Ledger does not exist - recreating... von-web_1 | von_generate_transactions -n von-web_1 | generate_indy_pool_transactions --nodes 4 --clients 0 --ips ,,, node2_1 | Generated genesis transaction file: node2_1 | /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis node2_1 | cat: /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis: No such file or directory node4_1 | Traceback (most recent call last): node4_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 253, in _bootstrap_args_type_ips_hosts node4_1 | ipaddress.ip_address(arg) node4_1 | File "/usr/lib/python3.5/ipaddress.py", line 54, in ip_address node4_1 | address) node4_1 | ValueError: '' does not appear to be an IPv4 or IPv6 address node4_1 | node4_1 | During handling of the above exception, another exception occurred: node4_1 | node4_1 | Traceback (most recent call last): node4_1 | File "/usr/local/bin/generate_indy_pool_transactions", line 19, in node4_1 | getTxnOrderedFields(), ConfigHelper, NodeConfigHelper) node4_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 197, in bootstrapTestNodes node4_1 | args = parser.parse_args() node4_1 | File "/usr/lib/python3.5/argparse.py", line 1735, in parse_args node4_1 | args, argv = self.parse_known_args(args, namespace) node4_1 | File "/usr/lib/python3.5/argparse.py", line 1767, in parse_known_args node4_1 | namespace, args = self._parse_known_args(args, namespace) node4_1 | File "/usr/lib/python3.5/argparse.py", line 1973, in _parse_known_args node4_1 | start_index = consume_optional(start_index) node4_1 | File "/usr/lib/python3.5/argparse.py", line 1913, in consume_optional node4_1 | take_action(action, args, option_string) node4_1 | File "/usr/lib/python3.5/argparse.py", line 1825, in take_action node4_1 | argument_values = self._get_values(action, argument_strings) node4_1 | File "/usr/lib/python3.5/argparse.py", line 2266, in _get_values node4_1 | value = self._get_value(action, arg_string) node4_1 | File "/usr/lib/python3.5/argparse.py", line 2295, in _get_value node4_1 | result = type_func(arg_string) node4_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 255, in _bootstrap_args_type_ips_hosts node4_1 | if not is_hostname_valid(arg): node4_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/util.py", line 538, in is_hostname_valid node4_1 | if hostname[-1] == ".": node4_1 | IndexError: string index out of range node4_1 | Generated genesis transaction file: node4_1 | /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis node4_1 | cat: /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis: No such file or directory node1_1 | Node-stack name is Node1 node1_1 | Client-stack name is Node1C node1_1 | Generating keys for random seed b'aCB03E7B1D10Ae3C0C4C9e9CDFDE6982' node1_1 | Init local keys for client-stack node1_1 | Public key is 805858382bd094f3327fe04c8569e2f8ffafb834e14eb5ed109de53320221841 node1_1 | Verification key is fdfbb6819c6beef7d1f7d475fe2b9745637460a0284bc3ad667bafb107d5c0ba node1_1 | Init local keys for node-stack node1_1 | Public key is 805858382bd094f3327fe04c8569e2f8ffafb834e14eb5ed109de53320221841 node1_1 | Verification key is fdfbb6819c6beef7d1f7d475fe2b9745637460a0284bc3ad667bafb107d5c0ba node1_1 | BLS Public key is 4iiehFk8KZsMsk27VBNepaaXMQgEgcXgxreWCDWNQVhrxiXy94b2si5r3UvFcVVX52GE96rjkhFrpqZW2wQs2wcaSYBFxZTSnFnwZYzaKZJEjDHAyR1gUmJtYtmPUBj9kmcXedALFiySt5M8ydWR23d6PCwAQBBRh8Sgp5GLh4HbHzx node1_1 | generate_indy_pool_transactions --nodes 4 --clients 0 --nodeNum 1 --ips ,,, von_node2_1 exited with code 1 node3_1 | Traceback (most recent call last): node3_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 253, in _bootstrap_args_type_ips_hosts node3_1 | ipaddress.ip_address(arg) node3_1 | File "/usr/lib/python3.5/ipaddress.py", line 54, in ip_address node3_1 | address) node3_1 | ValueError: '' does not appear to be an IPv4 or IPv6 address node3_1 | node3_1 | During handling of the above exception, another exception occurred: node3_1 | node3_1 | Traceback (most recent call last): node3_1 | File "/usr/local/bin/generate_indy_pool_transactions", line 19, in node3_1 | getTxnOrderedFields(), ConfigHelper, NodeConfigHelper) node3_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 197, in bootstrapTestNodes node3_1 | args = parser.parse_args() node3_1 | File "/usr/lib/python3.5/argparse.py", line 1735, in parse_args node3_1 | args, argv = self.parse_known_args(args, namespace) node3_1 | File "/usr/lib/python3.5/argparse.py", line 1767, in parse_known_args node3_1 | namespace, args = self._parse_known_args(args, namespace) node3_1 | File "/usr/lib/python3.5/argparse.py", line 1973, in _parse_known_args node3_1 | start_index = consume_optional(start_index) node3_1 | File "/usr/lib/python3.5/argparse.py", line 1913, in consume_optional node3_1 | take_action(action, args, option_string) node3_1 | File "/usr/lib/python3.5/argparse.py", line 1825, in take_action node3_1 | argument_values = self._get_values(action, argument_strings) node3_1 | File "/usr/lib/python3.5/argparse.py", line 2266, in _get_values node3_1 | value = self._get_value(action, arg_string) node3_1 | File "/usr/lib/python3.5/argparse.py", line 2295, in _get_value node3_1 | result = type_func(arg_string) node3_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 255, in _bootstrap_args_type_ips_hosts node3_1 | if not is_hostname_valid(arg): node3_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/util.py", line 538, in is_hostname_valid node3_1 | if hostname[-1] == ".": node3_1 | IndexError: string index out of range node3_1 | Generated genesis transaction file: node3_1 | /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis node3_1 | cat: /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis: No such file or directory von_node4_1 exited with code 1 node1_1 | Traceback (most recent call last): node1_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 253, in _bootstrap_args_type_ips_hosts node1_1 | ipaddress.ip_address(arg) node1_1 | File "/usr/lib/python3.5/ipaddress.py", line 54, in ip_address node1_1 | address) node1_1 | ValueError: '' does not appear to be an IPv4 or IPv6 address node1_1 | node1_1 | During handling of the above exception, another exception occurred: node1_1 | node1_1 | Traceback (most recent call last): node1_1 | File "/usr/local/bin/generate_indy_pool_transactions", line 19, in node1_1 | getTxnOrderedFields(), ConfigHelper, NodeConfigHelper) node1_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 197, in bootstrapTestNodes node1_1 | args = parser.parse_args() node1_1 | File "/usr/lib/python3.5/argparse.py", line 1735, in parse_args node1_1 | args, argv = self.parse_known_args(args, namespace) node1_1 | File "/usr/lib/python3.5/argparse.py", line 1767, in parse_known_args node1_1 | namespace, args = self._parse_known_args(args, namespace) node1_1 | File "/usr/lib/python3.5/argparse.py", line 1973, in _parse_known_args node1_1 | start_index = consume_optional(start_index) node1_1 | File "/usr/lib/python3.5/argparse.py", line 1913, in consume_optional node1_1 | take_action(action, args, option_string) node1_1 | File "/usr/lib/python3.5/argparse.py", line 1825, in take_action node1_1 | argument_values = self._get_values(action, argument_strings) node1_1 | File "/usr/lib/python3.5/argparse.py", line 2266, in _get_values node1_1 | value = self._get_value(action, arg_string) node1_1 | File "/usr/lib/python3.5/argparse.py", line 2295, in _get_value node1_1 | result = type_func(arg_string) node1_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 255, in _bootstrap_args_type_ips_hosts node1_1 | if not is_hostname_valid(arg): node1_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/util.py", line 538, in is_hostname_valid node1_1 | if hostname[-1] == ".": node1_1 | IndexError: string index out of range von-web_1 | Traceback (most recent call last): von-web_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 253, in _bootstrap_args_type_ips_hosts von-web_1 | ipaddress.ip_address(arg) von-web_1 | File "/usr/lib/python3.5/ipaddress.py", line 54, in ip_address von-web_1 | address) von-web_1 | ValueError: '' does not appear to be an IPv4 or IPv6 address von-web_1 | von-web_1 | During handling of the above exception, another exception occurred: von-web_1 | von-web_1 | Traceback (most recent call last): von-web_1 | File "/usr/local/bin/generate_indy_pool_transactions", line 19, in von-web_1 | getTxnOrderedFields(), ConfigHelper, NodeConfigHelper) von-web_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 197, in bootstrapTestNodes von-web_1 | args = parser.parse_args() von-web_1 | File "/usr/lib/python3.5/argparse.py", line 1735, in parse_args von-web_1 | args, argv = self.parse_known_args(args, namespace) von-web_1 | File "/usr/lib/python3.5/argparse.py", line 1767, in parse_known_args von-web_1 | namespace, args = self._parse_known_args(args, namespace) von-web_1 | File "/usr/lib/python3.5/argparse.py", line 1973, in _parse_known_args von-web_1 | start_index = consume_optional(start_index) von-web_1 | File "/usr/lib/python3.5/argparse.py", line 1913, in consume_optional von-web_1 | take_action(action, args, option_string) von-web_1 | File "/usr/lib/python3.5/argparse.py", line 1825, in take_action von-web_1 | argument_values = self._get_values(action, argument_strings) von-web_1 | File "/usr/lib/python3.5/argparse.py", line 2266, in _get_values von-web_1 | value = self._get_value(action, arg_string) von-web_1 | File "/usr/lib/python3.5/argparse.py", line 2295, in _get_value von-web_1 | result = type_func(arg_string) von-web_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/test_network_setup.py", line 255, in _bootstrap_args_type_ips_hosts von-web_1 | if not is_hostname_valid(arg): von-web_1 | File "/usr/local/lib/python3.5/dist-packages/plenum/common/util.py", line 538, in is_hostname_valid von-web_1 | if hostname[-1] == ".": von-web_1 | IndexError: string index out of range node1_1 | Generated genesis transaction file: node1_1 | /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis node1_1 | cat: /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis: No such file or directory von-web_1 | Generated genesis transaction file: von-web_1 | /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis von-web_1 | cat: /home/indy/.indy-cli/networks/sandbox/pool_transactions_genesis: No such file or directory von_node3_1 exited with code 1 von_node1_1 exited with code 1 von_von-web_1 exited with code 1

[Qingbao]

These logs came from docker for window 10, while works fine when changed to Ubuntu 16.04

[masterDev1985]

@Qingbao The network setup scripts try to get an IP address that the nodes can use to find each other. This IP address is then used to generate the genesis transaction for a pool ledger on each node. In our scripts, this IP is parsed from an ifconfig command. I'm betting that this command doesn't return anything on Windows 10, as Windows runs docker in a linux virtual machine. I'd test some of this myself, but I don't have a Windows 10 machine handy.

I think you could fix this on Win10 by setting the IP environment variable to the IP address of the docker virtual machine before running the manage script.

[herveblanc]

Hi Dale, What is the IPS environment variable used for? I get both warnings at run time and similarly end up with nodes into errors. Thx

[masterDev1985]

The IP and IPS environment variables are a way of manually specifying the IP addresses that should be passed to the nodes for them to be able to find each other. They let you specify a single IP address at which all the nodes may be reached, or a list of IP addresses where the individual nodes may be reached, respectively. You'd only need these env variables if the startup scripts were unable to automatically discover the IP address at which the nodes could reach each other.

[herveblanc]

Thx @masterDev1985 for your inputs. It looks to me like I shouldn't have to set IP or IPS if I'm running local on ubuntu native machine. Yet, run phase fails:

~/git/indy-ssivc-tutorial/von-network$ ./manage start WARNING: The IP variable is not set. Defaulting to a blank string. WARNING: The IPS variable is not set. Defaulting to a blank string. Removing von_node1_1 Removing von_node2_1 Removing von_node3_1 Removing von_node4_1 Recreating c2fdbe70eadd_von_node3_1 ... Recreating 63703de71d68_von_node2_1 ... error Recreating 35c6e809ee1e_von_node1_1 ... Recreating 8eeada11ed09_von_node4_1 ...

ERROR: for 63703de71d68_von_node2_1 Cannot start service node2: driver failed programming external connectivitRecreating 35c6e809ee1e_von_node1_1 ... error 9704 failed: port is already allocated Recreating c2fdbe70eadd_von_node3_1 ... error ERROR: for 35c6e809ee1e_von_node1_1 Cannot start service node1: driver failed programming external connectivity on endpoint von_node1_1 (b0d0dace06660035164da09a5184d692bb38874681a9e3ce22d59d46cdddc62a): Bind for 0.0.0.0:9702 failed: port is already allocated

ERROR: for c2fdbe70eadd_von_node3_1 Cannot start service node3: driver failed programming external connectivity on endpoint von_node3_1 (02a24dc26cd548a1df2f82d8cc80ee2ef6871504990e7426e4201bd74576f857): Bind for 0.0.0.0:Recreating 8eeada11ed09_von_node4_1 ... error

ERROR: for 8eeada11ed09_von_node4_1 Cannot start service node4: driver failed programming external connectivity on endpoint von_node4_1 (fc086f159d6df93f5f0589605d8583fa1743b1ee34c3c42e94745be00d9a34b9): Bind for 0.0.0.0:9708 failed: port is already allocated

ERROR: for node2 Cannot start service node2: driver failed programming external connectivity on endpoint von_node2_1 (94e67c18103e8262a86569818bdeb430da77267705029f18a6b21cbf362b070d): Bind for 0.0.0.0:9704 failed: port is already allocated

ERROR: for node1 Cannot start service node1: driver failed programming external connectivity on endpoint von_node1_1 (b0d0dace06660035164da09a5184d692bb38874681a9e3ce22d59d46cdddc62a): Bind for 0.0.0.0:9702 failed: port is already allocated

ERROR: for node3 Cannot start service node3: driver failed programming external connectivity on endpoint von_node3_1 (02a24dc26cd548a1df2f82d8cc80ee2ef6871504990e7426e4201bd74576f857): Bind for 0.0.0.0:9706 failed: port is already allocated

ERROR: for node4 Cannot start service node4: driver failed programming external connectivity on endpoint von_node4_1 (fc086f159d6df93f5f0589605d8583fa1743b1ee34c3c42e94745be00d9a34b9): Bind for 0.0.0.0:9708 failed: port is already allocated ERROR: Encountered errors while bringing up the project.

[herveblanc]

Sorry for my previous noisy post, I had leftover docker instances locking port usage. sudo service docker stop then sudo service docker start got things back in order.

[masterDev1985]

Ah, that'll do it! FYI there are ./manage stop and ./manage rm commands for von-network, permitify, and TheOrgBook that should clean out old containers. They're probably more convenient than restarting docker.

Haven't heard from @Qingbao on his issue, so I'm going to close this one or get @brycecurtis to close it.