[dev_network] set_primary_validator failed due to Connection refused

[DucPhamTV]

Bug Description set_primary_validator failed due to Connection refused

Steps to Reproduce update .env, set PUBLIC_IP_ADDRESS=127.0.0.1 docker-compose up

$ docker-compose up
bank_1         | Waiting for PV... 
pv_1           | HTTP GET /config 200 [0.29, 172.19.0.5:56118]                                                                                                                                   
bank_1         | PV started
bank_1         | Bank initialization complete                                                                                                                                                    
bank_1         | Waiting for self...
bank_1         | HTTP GET /config 200 [0.47, 127.0.0.1:47972]
bank_1         | self started

But at step set_primary_validator, it raised the error:

...
bank_1         | During handling of the above exception, another exception occurred:                                                                                                             
bank_1         | 
bank_1         | Traceback (most recent call last):
bank_1         |   File "/usr/local/lib/python3.8/site-packages/thenewboston/base_classes/fetch_primary_validator_config.py", line 101, in handle
bank_1         |     config = self.fetch_validator_config()
bank_1         |   File "/usr/local/lib/python3.8/site-packages/thenewboston/base_classes/fetch_primary_validator_config.py", line 82, in fetch_validator_config
bank_1         |     results = fetch(url=url, headers={})
bank_1         |   File "/usr/local/lib/python3.8/site-packages/thenewboston/utils/network.py", line 8, in fetch
bank_1         |     response = requests.get(url, headers=headers)
bank_1         |   File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 76, in get
bank_1         |     return request('get', url, params=params, **kwargs)
bank_1         |   File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
bank_1         |     return session.request(method=method, url=url, **kwargs)
bank_1         |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
bank_1         |     resp = self.send(prep, **send_kwargs)
bank_1         |   File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
bank_1         |     r = adapter.send(request, **kwargs)
bank_1         |   File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
bank_1         |     raise ConnectionError(e, request=request)
bank_1         | requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8001): Max retries exceeded with url: /config (Caused by NewConnectionError('<urllib3.connection.
HTTPConnection object at 0x7f92bbe81100>: Failed to establish a new connection: [Errno 111] Connection refused'))

Expected behavior other containers can get config from PV.

Actual behavior Connection refuse when getting config from PV.

OS and Browser Ubuntu-20.04 docker-compose version 1.25.0 docker version 19.03.8,

[DucPhamTV]

127.0.0.1:8001 is unreachable inside a container. but pv:8000 is reachable.

$ docker exec -it bank_bank_1 curl http://pv:8000 -v
*   Trying 172.19.0.4:8000...
* Connected to pv (172.19.0.4) port 8000 (#0)
> GET / HTTP/1.1
> Host: pv:8000
> User-Agent: curl/7.69.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Vary: Accept, Cookie, Origin
< Allow: GET, HEAD, OPTIONS
< Server-Timing: TimerPanel_utime;dur=7.183999999999635;desc="User CPU time", TimerPanel_stime;dur=3.8770000000000193;desc="System CPU time", TimerPanel_total;dur=11.060999999999655;desc="Total CPU time", TimerPanel_total_time;dur=11.115074157714844;desc="Elapsed time", SQLPanel_sql_time;dur=0;desc="SQL 0 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
< X-Frame-Options: DENY
< Content-Length: 575
< X-Content-Type-Options: nosniff
< Referrer-Policy: same-origin
< 
* Connection #0 to host pv left intact
{"accounts":"http://pv:8000/accounts","bank_blocks":"http://pv:8000/bank_blocks","bank_confirmation_services":"http://pv:8000/bank_confirmation_services","banks":"http://pv:8000/banks","clean":"http://pv:8000/clean","confirmation_blocks":"http://pv:8000/confirmation_blocks","crawl":"http://pv:8000/crawl","config":"http://pv:8000/config","primary_validator_updated":"http://pv:8000/primary_validator_updated","upgrade_request":"http://pv:8000/upgrade_request","confirmation_block_history":"http://pv:8000/confirmation_block_history","validators":"http://pv:8000/validators"}

And

$ docker exec -it bank_bank_1 curl http://127.0.0.1:8001 -v
*   Trying 127.0.0.1:8001...
* connect to 127.0.0.1 port 8001 failed: Connection refused
* Failed to connect to 127.0.0.1 port 8001: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 8001: Connection refused

I'm thinking about using docker network IP instead of PUBLIC_IP_ADDRESS=127.0.0.1 ...

[vosi]

@DucPhamTV probably something is wrong with docker proxy or some other process is listening on 8001 port ...

also, what I would like to see

delete exsiting volumes

start docker-compose up
wait for couple of mins, and copy the ENTIRE log and attach here as a file

also, on you host machine do netstat -tap and paste a responce here

[vosi]

also docker exec -it bank_bank_1 curl http://127.0.0.1:8001 -v is wrong
this cmd should be run from host machine curl http://127.0.0.1:8001

[vosi]

@DucPhamTV fixed, try again

[DucPhamTV]

@vosi .It still doesn't work for me. I don't see new update on master branch related to this issue. Just a new commit about update database names. Here is log when docker compose up is provisioned docker-compose-log

Here is netstat -tap. filtered unrelated information.

tcp6       0      0 [::]:8001               [::]:*                  LISTEN      -                   
tcp6       0      0 [::]:8002               [::]:*                  LISTEN      -                   
tcp6       0      0 [::]:8003               [::]:*                  LISTEN      -                   
tcp6       0      0 [::]:8004               [::]:*                  LISTEN      -  

Only PV listens on 8001. I can access it by my browser.

also docker exec -it bank_bank_1 curl http://127.0.0.1:8001 -v is wrong

AFAIK, What exactly set_primary_validator is doing this, except that it uses python requests, I used curl. I know inside a container, It can't reach 127.0.0.1:8001 on host. Since .env, set PUBLIC_IP_ADDRESS=127.0.0.1 will cause this thing happen.

[vosi]

ah .. .right, the thing is, my docker setup is on a dedicated server, haven't tried provisioning the network on a setup with a local docker host I suppose a simple fix would be to use some 10/8 or 172.16/12 or 192.168/16 network

I'll update the doc shortly