search

status-im / infra-status-legacy

Infrastructure for old Status fleet
https://github.com/status-im/nim-waku
1 stars 3 forks source link

Enable websocket support for `status.*` fleets #13

Closed felicio closed 2 years ago

felicio commented 2 years ago

What

Why

jakubgs commented 2 years ago

Changes:

Done:

 > c fleets.status.im | jq '.fleets."status.test"."wss/p2p/waku"'

{
  "node-01.ac-cn-hongkong-c.status.test": "/dns4/node-01.ac-cn-hongkong-c.status.test.statusim.net/tcp/443/wss/p2p/16Uiu2HAm2BjXxCp1sYFJQKpLLbPbwd5juxbsYofu3TsS3auvT9Yi",
  "node-01.do-ams3.status.test": "/dns4/node-01.do-ams3.status.test.statusim.net/tcp/443/wss/p2p/16Uiu2HAkukebeXjTQ9QDBeNDWuGfbaSg79wkkhK4vPocLgR6QFDf",
  "node-01.gc-us-central1-a.status.test": "/dns4/node-01.gc-us-central1-a.status.test.statusim.net/tcp/443/wss/p2p/16Uiu2HAmGDX3iAFox93PupVYaHa88kULGqMpJ7AEHGwj3jbMtt76"
}
 > c fleets.status.im | jq '.fleets."status.prod"."wss/p2p/waku"'
{
  "node-01.ac-cn-hongkong-c.status.prod": "/dns4/node-01.ac-cn-hongkong-c.status.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC",
  "node-01.do-ams3.status.prod": "/dns4/node-01.do-ams3.status.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D",
  "node-01.gc-us-central1-a.status.prod": "/dns4/node-01.gc-us-central1-a.status.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb",
  "node-02.ac-cn-hongkong-c.status.prod": "/dns4/node-02.ac-cn-hongkong-c.status.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8",
  "node-02.do-ams3.status.prod": "/dns4/node-02.do-ams3.status.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV",
  "node-02.gc-us-central1-a.status.prod": "/dns4/node-02.gc-us-central1-a.status.prod.statusim.net/tcp/443/wss/p2p/16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg"
}
prichodko commented 2 years ago

@jakubgs thanks for making this happen. However, it looks that only this node (/dns4/node-01.ac-cn-hongkong-c.status.test.statusim.net/tcp/443/wss/p2p/16Uiu2HAm2BjXxCp1sYFJQKpLLbPbwd5juxbsYofu3TsS3auvT9Yi) is working. The rest of them fail on the WebSocket is closed before the connection is established error when trying to connect.

Let me know if you need more info.

jakubgs commented 2 years ago

Indeed, some ports appear to be filtered:

 > sudo nmap -Pn -p443 node-01.ac-cn-hongkong-c.status.prod node-01.do-ams3.status.prod node-01.gc-us-central1-a.status.prod node-02.ac-cn-hongkong-c.status.prod node-02.do-ams3.status.prod node-02.gc-us-central1-a.status.prod | grep -E '(report|tcp)'     
Nmap scan report for node-01.ac-cn-hongkong-c.status.prod (47.242.202.59)
443/tcp filtered https
Nmap scan report for node-01.do-ams3.status.prod (143.198.249.174)
443/tcp filtered https
Nmap scan report for node-01.gc-us-central1-a.status.prod (35.202.55.153)
443/tcp open  https
Nmap scan report for node-02.ac-cn-hongkong-c.status.prod (47.243.128.134)
443/tcp filtered https
Nmap scan report for node-02.do-ams3.status.prod (161.35.244.121)
443/tcp filtered https
Nmap scan report for node-02.gc-us-central1-a.status.prod (34.132.213.233)
443/tcp open  https
jakubgs commented 2 years ago

And indeed, looks like some ports were not properly open. Maybe I missed running open-ports for some hosts. Not it's good:

 > sudo nmap -Pn -p443 node-01.ac-cn-hongkong-c.status.prod node-01.do-ams3.status.prod node-01.gc-us-central1-a.status.prod node-02.ac-cn-hongkong-c.status.prod node-02.do-ams3.status.prod node-02.gc-us-central1-a.status.prod | grep -E '(report|tcp)'
Nmap scan report for node-01.ac-cn-hongkong-c.status.prod (47.242.202.59)
443/tcp open  https
Nmap scan report for node-01.do-ams3.status.prod (143.198.249.174)
443/tcp open  https
Nmap scan report for node-01.gc-us-central1-a.status.prod (35.202.55.153)
443/tcp open  https
Nmap scan report for node-02.ac-cn-hongkong-c.status.prod (47.243.128.134)
443/tcp open  https
Nmap scan report for node-02.do-ams3.status.prod (161.35.244.121)
443/tcp filtered https
Nmap scan report for node-02.gc-us-central1-a.status.prod (34.132.213.233)
443/tcp open  https
felicio commented 2 years ago

For future reference, even after re/opening the ports, and even though not reproducible at the moment, connecting to some peers continued to fail randomly.

Logs from such failed connection:

image https://kibana.infra.status.im/goto/eb488550-249c-11ed-b14f-29f692784ff2

Cc @fryorcraken @jm-clius

fryorcraken commented 2 years ago

@felicio I would expected to see some errors in the nwaku nodes as any peer on the internet can now started to negotiate an http/https request on this port (and fail).

When you see the issue, it would be valuable to record in an issue the local peer logs (whether it's js-waku, go-waku, etc) with timestamps so we can investigate the issue from both side and pull the relevant logs from the fleet nodes.