search

IntersectMBO / cardano-node

The core component that is used to participate in a Cardano decentralised blockchain.
https://cardano.org
Apache License 2.0
3.06k stars 720 forks source link

[BUG] - DNS Address fails while IP address is working #2752

Closed toky03 closed 3 years ago

toky03 commented 3 years ago

External

Area Other Cardano Node Relay Connection

Summary A clear and specific description of what the bug is.

Steps to reproduce Start a cardano node inside of a docker container e.g:

cardano-node run \
   --topology path/to/mainnet-topology.json \
   --database-path path/to/db \
   --socket-path path/to/db/node.socket \
   --host-addr 0.0.0.0 \
   --port 3001 \
   --config path/to/mainnet-config.json

while the topology.json files has an entry with a dns name (not an ip address)

{
  "Producers": [
    {
      "addr": "relays-new.cardano-testnet.iohkdev.io",
      "port": 3001,
      "valency": 2
    }
  ]
}

With this configuration it fails:

[seeland-:cardano.node.dns-producers:Notice:5] [2021-05-26 19:08:49.68 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-testnet.iohkdev.io", dstPort = 3001, dstValency = 2}]
[seeland-:cardano.node.ip-producers:Notice:5] [2021-05-26 19:08:49.68 UTC] IPSubscriptionTarget {ispIps = [], ispValency = 0}
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened imm db with immutable tip at genesis (origin) and chunk 0
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened vol db
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Replaying ledger from genesis
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened lgr db
[seeland-:cardano.node.ChainDB:Info:5] [2021-05-26 19:08:49.72 UTC] Opened db with immutable tip at genesis (origin) and tip genesis (origin)
[seeland-:cardano.node.ChainDB:Notice:34] [2021-05-26 19:08:49.73 UTC] Chain extended, new tip: 1e64e74bd7ac76d6806480a28017deb0aedd356fb61844ec95c429ae2f30c7c3 at slot 0
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:08:49.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:08:59.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:09.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.ChainDB:Notice:76] [2021-05-26 19:09:18.86 UTC] before next, messages elided = 2259510500755060
[seeland-:cardano.node.ChainDB:Notice:76] [2021-05-26 19:09:18.86 UTC] Chain extended, new tip: 1e64e74bd7ac76d6806480a28017deb0aedd356fb61844ec95c429ae2f30c7c3 at slot 0
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:19.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[seeland-:cardano.node.DnsSubscription:Warning:55] [2021-05-26 19:09:29.73 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions

If you use an ip address instead it works.

I checked if there is an issue with Kubernetes and DNS but i am able to resolve the dns from within the pod:

root@seeland-stakepool-relay1-77587998b4-zj6k5:/config# nslookup relays-new.cardano-testnet.iohkdev.io
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 54.250.11.33
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.129.133.68
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.9.80.183
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.124.178.139
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 175.41.160.108
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 18.159.64.253
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 54.241.77.32
Name:   relays-new.cardano-testnet.iohkdev.io
Address: 3.128.220.55

Expected behavior DNSSubscription should work like it works for IP addresses.

System info:

Additional context The Problem has been disscussed in the Cardano Forum DNS Address fails while IP Address works

DiazRock commented 3 years ago

Got the same here. Solved using ip. This is my system info:

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

docker version

Docker version 20.10.2, build 20.10.2-0ubuntu1~18.04.2

I'm using microk8s

ghost commented 3 years ago

Same problem here. The node syncs on the mainnet for some time but then fails. I have to delete the state folder /db for the node to restart synchronization from the start.

weebl2000 commented 3 years ago

I can try to have a look if I can reproduce & try to fix.

weebl2000 commented 3 years ago

Tried to reproduce but couldn't. More specific reproduction steps?

toky03 commented 3 years ago

Hi @weebl2000 I built the Docker Cardano node with the following Dockerfile:

FROM haskell:latest as cardano

RUN cabal update
RUN apt-get update -y
RUN apt-get install -y git libtool build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 -y

RUN git clone https://github.com/input-output-hk/libsodium

WORKDIR libsodium
RUN git checkout 66f017f1 && ./autogen.sh && ./configure &&  make
RUN make install

ENV LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
ENV PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"
ENV VERSION_TAG=1.27.0

RUN mkdir /cardano
WORKDIR /cardano
RUN git clone https://github.com/input-output-hk/cardano-node.git
WORKDIR cardano-node
RUN  git fetch --all --recurse-submodules --tags && git tag && git checkout tags/$VERSION_TAG
RUN cabal configure --with-compiler=ghc-8.10.2
RUN echo  "package cardano-crypto-praos" >>  cabal.project.local && echo "  flags: -external-libsodium-vrf" >>  cabal.project.local

RUN cabal build all

RUN cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-node-$VERSION_TAG/x/cardano-node/build/cardano-node/cardano-node /bin/cardano-node
RUN cp -p dist-newstyle/build/x86_64-linux/ghc-8.10.2/cardano-cli-$VERSION_TAG/x/cardano-cli/build/cardano-cli/cardano-cli /bin/cardano-cli

RUN chmod +x /bin/cardano-cli
RUN /bin/cardano-cli --version

FROM ubuntu:latest

RUN apt update

COPY --from=cardano /usr/local/ /usr/local

ENV LD_LIBRARY_PATH="/usr/local/lib:$LD_LIBRARY_PATH"
ENV PKG_CONFIG_PATH="/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH"

COPY --from=cardano /bin /usr/bin

RUN chmod +x /usr/bin/cardano-cli
RUN mkdir /database

COPY --from=cardano /usr/local/ /usr/local

WORKDIR /config

then i downloaded all the configuration files inside of a data folder

wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-config.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-byron-genesis.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-shelley-genesis.json
wget https://hydra.iohk.io/job/Cardano/cardano-node/cardano-deployment/latest-finished/download/1/testnet-topology.json

Build the docker Image (this takes quite some time as it builds the cardano node from source)

docker build -t cardano-node-image:test .

run the docker container

docker container run -it -v $(pwd)/data/:/config cardano-node-image:test /bin/bash

Now inside of the docker container run the node:

./cardano-node run --topology testnet-topology.json --database-path db --socket-path db/node.socket --config testnet-config.json --port 3000 --host-addr 0.0.0.0

weebl2000 commented 3 years ago

Are you sure it's complete? When running I get:

cabal: Cannot find the program 'ghc'. User-specified path 'ghc-8.10.4' does not refer to an executable and the program is not on the system path.

toky03 commented 3 years ago

I just checked it and i had to use this command instead: /usr/bin/cardano-node run --topology testnet-topology.json --database-path db --socket-path db/node.socket --config testnet-config.json --port 3000 --host-addr 0.0.0.0 Did you try to build the Docker Image with the Dockerfile given above? If you want to try it with a pre built Docker Image (then you do not have to build it again) you can use: docker pull toky03/cardano-node-image:test and then run the container with: docker container run -it -v $(pwd)/data/:/config toky03/cardano-node-image:test /bin/bash With this i always get the same result. [85f5d4cc:cardano.node.DnsSubscription:Warning:55] [2021-06-07 13:17:30.89 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions Do you use another Docker Cardano Image which works?

weebl2000 commented 3 years ago

I've tried with inputoutput/cardano-node image, which seems to work:

My /tmp/cardano just contains config and a "db" folder which is empty.

$ docker run -v /tmp/cardano:/tmp/cardano  inputoutput/cardano-node run --topology /tmp/cardano/mainnet-topology.json --database-path /tmp/cardano/db --socket-path /tmp/cardano/db/node.socket --host-addr 0.0.0.0 --port 3001 --config /tmp/cardano/mainnet-config.json
Listening on http://127.0.0.1:12798
[26221b64:cardano.node.networkMagic:Notice:5] [2021-06-07 13:26:50.90 UTC] NetworkMagic 764824073
[26221b64:cardano.node.basicInfo.protocol:Notice:5] [2021-06-07 13:26:50.90 UTC] Byron; Shelley
[26221b64:cardano.node.basicInfo.version:Notice:5] [2021-06-07 13:26:50.90 UTC] 1.27.0
[26221b64:cardano.node.basicInfo.commit:Notice:5] [2021-06-07 13:26:50.90 UTC] 8fe46140a52810b6ca456be01d652ca08fe730bf
[26221b64:cardano.node.basicInfo.nodeStartTime:Notice:5] [2021-06-07 13:26:50.90 UTC] 2021-06-07 13:26:50.900762829 UTC
[26221b64:cardano.node.basicInfo.systemStartTime:Notice:5] [2021-06-07 13:26:50.90 UTC] 2017-09-23 21:44:51 UTC
[26221b64:cardano.node.basicInfo.slotLengthByron:Notice:5] [2021-06-07 13:26:50.90 UTC] 20s
[26221b64:cardano.node.basicInfo.epochLengthByron:Notice:5] [2021-06-07 13:26:50.90 UTC] 21600
[26221b64:cardano.node.basicInfo.slotLengthShelley:Notice:5] [2021-06-07 13:26:50.90 UTC] 1s
[26221b64:cardano.node.basicInfo.epochLengthShelley:Notice:5] [2021-06-07 13:26:50.90 UTC] 432000
[26221b64:cardano.node.basicInfo.slotsPerKESPeriodShelley:Notice:5] [2021-06-07 13:26:50.90 UTC] 129600
[26221b64:cardano.node.basicInfo.slotLengthAllegra:Notice:5] [2021-06-07 13:26:50.90 UTC] 1s
[26221b64:cardano.node.basicInfo.epochLengthAllegra:Notice:5] [2021-06-07 13:26:50.90 UTC] 432000
[26221b64:cardano.node.basicInfo.slotsPerKESPeriodAllegra:Notice:5] [2021-06-07 13:26:50.90 UTC] 129600
[26221b64:cardano.node.basicInfo.slotLengthMary:Notice:5] [2021-06-07 13:26:50.90 UTC] 1s
[26221b64:cardano.node.basicInfo.epochLengthMary:Notice:5] [2021-06-07 13:26:50.90 UTC] 432000
[26221b64:cardano.node.basicInfo.slotsPerKESPeriodMary:Notice:5] [2021-06-07 13:26:50.90 UTC] 129600
[26221b64:cardano.node.addresses:Notice:5] [2021-06-07 13:26:50.90 UTC] [SocketInfo 0.0.0.0:3001]
[26221b64:cardano.node.diffusion-mode:Notice:5] [2021-06-07 13:26:50.90 UTC] InitiatorAndResponderDiffusionMode
[26221b64:cardano.node.dns-producers:Notice:5] [2021-06-07 13:26:50.90 UTC] [DnsSubscriptionTarget {dstDomain = "relays-new.cardano-mainnet.iohk.io", dstPort = 3001, dstValency = 2}]
[26221b64:cardano.node.ip-producers:Notice:5] [2021-06-07 13:26:50.90 UTC] IPSubscriptionTarget {ispIps = [], ispValency = 0}
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.16 UTC] Opened imm db with immutable tip at genesis (origin) and chunk 0
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.16 UTC] Opened vol db
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.16 UTC] Replaying ledger from genesis
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.29 UTC] Opened lgr db
[26221b64:cardano.node.ChainDB:Info:5] [2021-06-07 13:26:51.29 UTC] Opened db with immutable tip at genesis (origin) and tip genesis (origin)
[26221b64:cardano.node.ChainDB:Notice:35] [2021-06-07 13:26:51.29 UTC] Chain extended, new tip: d2d90d10993b3eb309f5485a5413cb9c9fe4f088e79b7b4e815dbbe8fcb8e4b7 at slot 0
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] CreatingServerSocket 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] ConfiguringServerSocket 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] ListeningServerSocket 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] ServerSocketUp 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:57] [2021-06-07 13:26:51.29 UTC] RunServer 0.0.0.0:3001
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] CreateSystemdSocketForSnocketPath "/tmp/cardano/db/node.socket"
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] CreatedLocalSocket "/tmp/cardano/db/node.socket"
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] ConfiguringLocalSocket "/tmp/cardano/db/node.socket" (FileDescriptor 25)
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] ListeningLocalSocket "/tmp/cardano/db/node.socket" (FileDescriptor 25)
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] LocalSocketUp "/tmp/cardano/db/node.socket" (FileDescriptor 25)
[26221b64:cardano.node.DiffusionInitializationTracer:Info:59] [2021-06-07 13:26:51.29 UTC] RunLocalServer (LocalAddress "/tmp/cardano/db/node.socket")
[26221b64:cardano.node.DnsSubscription:Notice:67] [2021-06-07 13:26:51.30 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 18.133.59.65:3001
[26221b64:cardano.node.DnsSubscription:Notice:67] [2021-06-07 13:26:51.32 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt End, destination 18.133.59.65:3001 outcome: ConnectSuccess
[26221b64:cardano.node.DnsSubscription:Notice:68] [2021-06-07 13:26:51.33 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 3.129.199.184:3001
[26221b64:cardano.node.DnsSubscription:Notice:82] [2021-06-07 13:26:51.35 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt Start, destination 18.159.65.82:3001
[26221b64:cardano.node.DnsSubscription:Notice:82] [2021-06-07 13:26:51.37 UTC] Domain: "relays-new.cardano-mainnet.iohk.io" Connection Attempt End, destination 18.159.65.82:3001 outcome: ConnectSuccessLast
[26221b64:cardano.node.ErrorPolicy:Notice:54] [2021-06-07 13:26:51.37 UTC] IP 3.129.199.184:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
[26221b64:cardano.node.ChainDB:Info:35] [2021-06-07 13:26:51.50 UTC] Block fits onto some fork: 89d9b5a5b8ddc8d7e5a6795e9774d97faf1efea59b2caf7eaf9f8c5b32059df4 at slot 0
[26221b64:cardano.node.ChainDB:Info:35] [2021-06-07 13:26:51.53 UTC] Block fits onto some fork: f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f at slot 0
[26221b64:cardano.node.ChainDB:Notice:35] [2021-06-07 13:26:51.54 UTC] Switched to a fork, new tip: f0f7892b5c333cffc4b3c4344de48af4cc63f55e44936196f365a9ef2244134f at slot 0
[26221b64:cardano.node.ChainDB:Notice:35] [2021-06-07 13:26:51.54 UTC] Chain extended, new tip: 1dbc81e3196ba4ab9dcb07e1c37bb28ae1c289c0707061f28b567c2f48698d50 at slot 1
toky03 commented 3 years ago

It works with the official inputoutput image indeed. i still have some issues with communicating via dns to my nodes via kubernetes service dns directly but i assume it is some misconficuration on my side.

Do you know by any chance how the official docker image is built differently? I am not familiar with the nix docker build inside the Repository

weebl2000 commented 3 years ago

I've never used a nix docker build like that, I did try to run it yesterday but did not succeed and did not put more time into it.

I did make a cardano-node build inside an ubuntu 20.04 container yesterday and that all seemed to work fine.

You could try enabling more verbose tracing by changing the following in the config:

  "TraceDNSResolver": true,
  "TraceDNSSubscription": true,
  "TraceDiffusionInitialization": true,
  "TracingVerbosity": "MaximalVerbosity",
  "minSeverity": "Debug",

You might want to disable some other logging since it will probably output a lot.

toky03 commented 3 years ago

Thank you for the hint @weebl2000. Unfortunately there is not nothing more which helps me to find the error:

[465fb394:cardano.node.DnsSubscription:Debug:55] [2021-06-11 09:23:27.59 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Starting Subscription Worker, valency 2
[465fb394:cardano.node.DnsSubscription:Warning:55] [2021-06-11 09:23:27.59 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Failed to start all required subscriptions
[465fb394:cardano.node.DnsSubscription:Debug:55] [2021-06-11 09:23:28.60 UTC] Domain: "relays-new.cardano-testnet.iohkdev.io" Restarting Subscription after 1.005226535s desired valency 2 current valency 0

Meanwhile i will use the official Cardano Image and try to figure out whats the Problem with the internal Kubernetes Service.

toky03 commented 3 years ago

I will close this issue as we can use the official cardano-node image which is working correctly.

Katochimoto commented 3 years ago

It works for me, I hope it helps you too smth like this in Dockerfile

RUN apt-get update && apt-get install -y netbase

From here https://github.com/input-output-hk/cardano-node/issues/953#issuecomment-636761169

DonaldJimenez commented 2 years ago

Just had the same issue while trying to start a fresh node on Windows 10. My node couldn't get past this error for a full day.

Domain: "relays-new.cardano-mainnet.iohk.io" Failed to start all required subscriptions

I modified the topology file and replaced domain name with an IP address. It immediately helped and node started to download blockchain data.

locallycompact commented 2 years ago

Also getting this error trtying to connect to the testnet

SebastienGllmt commented 2 years ago

I ran into this issue as well and it turns out the DNS lookup on my machine was having issues (running nslookup relays-new.cardano-mainnet.iohk.io would return "REFUSED")

Fixed this by changing my DNS settings on my machine

TypicalHog commented 2 years ago

I semi-fixed it by changing the mainnet-topology.json to this:

{
  "Producers": [
    {
      "addr": "3.127.133.53",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.194.25.1",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.159.63.239",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "3.22.52.164",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.133.108.207",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "18.180.136.78",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "54.176.88.11",
      "port": 3001,
      "valency": 2
    },
    {
      "addr": "54.238.30.120",
      "port": 3001,
      "valency": 2
    }
  ]
}

I got the IPs from nslookup relays-new.cardano-mainnet.iohk.io

EDIT: I do get errors every now and then tho, as some of the IPs seem to not work. log.txt