planetarium / NineChronicles.Headless

A headless node of NineChronicles game network, powered by Libplanet.
https://planetarium.github.io/NineChronicles.Headless/
GNU Affero General Public License v3.0
36 stars 41 forks source link

TURN relay server settings are ignored in `--libplanet-node` #200

Closed longfin closed 3 years ago

longfin commented 3 years ago
--libplanet-node \
--workers=50 \
-V="100021/6ec8E598962F1f475504F82fD5bF3410eAE58B9B/MEQCIE0NunabTpYPEiaj4PqO1ua4PmbgVNDf50fVPKHYmuExAiByZpCsgom+Gtm3Er+Eb4J3mp2Nk4+ilAD40+wq8YM40g==/ZHUxNjpXaW5kb3dzQmluYXJ5VXJsdTU2Omh0dHBzOi8vZG93bmxvYWQubmluZS1jaHJvbmljbGVzLmNvbS92MTAwMDIxL1dpbmRvd3MuemlwdTE0Om1hY09TQmluYXJ5VXJsdTU3Omh0dHBzOi8vZG93bmxvYWQubmluZS1jaHJvbmljbGVzLmNvbS92MTAwMDIxL21hY09TLnRhci5nenU5OnRpbWVzdGFtcHUxMDoyMDIwLTExLTI3ZQ==" \
--trusted-app-protocol-version-signer=03eeedcd574708681afb3f02fb2aef7c643583089267d17af35e978ecaf2a1184e \
-G=https://9c-test.s3.ap-northeast-2.amazonaws.com/genesis-block-9c-main \
--port=31234 \
--store-path=/app/data/9c-main \
--store-type=rocksdb \
-I="turn://0ed3e48007413e7c2e638f13ddd75ad272c6c507e081bd76a75e4b7adc86c9af:0apejou+ycZFfwtREeXFKdfLj2gCclKzz5ZJ49Cmy6I=@turn-us.planetarium.dev:3478" \
-I="turn://0ed3e48007413e7c2e638f13ddd75ad272c6c507e081bd76a75e4b7adc86c9af:0apejou+ycZFfwtREeXFKdfLj2gCclKzz5ZJ49Cmy6I=@turn-us1.planetarium.dev:3478" \
-I="turn://0ed3e48007413e7c2e638f13ddd75ad272c6c507e081bd76a75e4b7adc86c9af:0apejou+ycZFfwtREeXFKdfLj2gCclKzz5ZJ49Cmy6I=@turn-us2.planetarium.dev:3478" \
--peer=027bd36895d68681290e570692ad3736750ceaab37be402442ffb203967f98f7b6,9c-main-seed-1.planetarium.dev,31234 \
--peer=02f164e3139e53eef2c17e52d99d343b8cbdb09eeed88af46c352b1c8be6329d71,9c-main-seed-2.planetarium.dev,31234 \
--peer=0247e289aa332260b99dfd50e578f779df9e6702d67e50848bb68f3e0737d9b9a5,9c-main-seed-3.planetarium.dev,31234

With the above options, NineChronicles.Standalone.Executable didn't seem to use the TURN relay server(--ice-server).

{"log":"[09:25:33 INF] Listen on 31234\n","stream":"stdout","time":"2020-11-30T09:25:33.467013884Z"}
{"log":"[09:25:33 DBG] Starting swarm...\n","stream":"stdout","time":"2020-11-30T09:25:33.468645177Z"}
{"log":"[09:25:33 DBG] Peer information : 0x0E7745D2e9553037f2B8c2A226Cb4F348920d221.Unspecified/0.0.0.0:31234.\n","stream":"stdout","time":"2020-11-30T09:25:33.469382688Z"}
area363 commented 3 years ago

Update to @longfin

This was an issue in the Dockerfile at: https://github.com/planetarium/NineChronicles.Standalone/blob/99a61c61f1a3a9aaf312ee6b87da648d081e6b56/Dockerfile#L34 in which the Docker image is built with a default host of "0.0.0.0". So if a port parameter is not given in the docker run command, the image takes its default host.

Removing ["host", "0.0.0.0" ] will take care of this issue; however, the greater issue is that the docker container is not working consistently when connecting to the TURN relay server and stops working after 15-min max. Specifically, receiving most recent blocks through TURN works initially, but after 5-10 minutes, connections to the TURN servers are disconnected. Here are some symptoms I recorded when comparing the two options (Docker vs. non-Docker).

A. As you can see below, this is the lsof list of 9c-headless when running without Docker (plain dotnet run execution). The port connection to the peers and ice-servers(nat-stun-port) are all "ESTABLISHED" and stable. Screen Shot 2020-12-07 at 11 09 27 AM

B. When running on the Docker container on the other hand, its connection to peers are stable, but the ice-server connections are often disconnected. Screen Shot 2020-12-07 at 11 05 03 AM

C. Interestingly, the lsof list inside the container shows connection to the peers but no connection to the ice-server. Screen Shot 2020-12-07 at 11 17 08 AM

This leads me to believe that:

  1. The loss of connection to the ice-server piling up eventually causes the container to stop working.
  2. For some reason, Docker container can't maintain connection to the TURN server (connect works in the beginning).

I feel that this is a porting issue when running the docker but the issue still occurs even when using the "--net=host" option. I'm tempted to think this might be a UDP issue but it's only a suspicion and need more evidence to support this thought.