search

celestiaorg / celestia-node

Celestia Data Availability Nodes
Apache License 2.0
935 stars 932 forks source link

Attempt to spin up a DA light client node fails when bind-mounting the celestia block dir from docker compose #2799

Closed appetrosyan closed 1 year ago

appetrosyan commented 1 year ago

Celestia Node version

ghcr.io/celestiaorg/celestia-app:v1.0.0-rc14

OS

Alpine linux

Install tools

Docker, with docker compose

Others

No response

Steps to reproduce it

  1. Create docker-compose.yml with the following contents

    
version: '3'
services:
celestia:
build: ./celestia
volumes:
 - ./.local_data/celestia:/home/celestia
ports:
  - "26657:26657"
  - "26658:26658"
  - "26659:26659"
  - "9090:9090"

#use ganache for now
eth:
# image: trufflesuite/ganache-cli:latest
build: ./eth
ports:
  - "8545:8545"
volumes:
  - ./.local_data/ganache_data:/ganache_data
entrypoint:
  - ganache
  - --database.dbPath=/ganache_data
  - -m
  - 'minimum symptom minute gloom tragic situate silver mechanic salad amused elite beef'
  - -h
  - '0.0.0.0'

# build is extremely slow inside docker on m1 mac, might still want to run this externally or just connect to our own devnet
# solana:
#   build: ./solana
#   ports:
#     - "8899:8899"
#     - "8900:8900"
#     - "8001:8001"
#   mem_limit: 8G

2. Create a ganache network via :

```dockerfile
FROM node:18-bullseye-slim as base

RUN apt-get update && \
    apt-get install --no-install-recommends -y \
        build-essential \
        python3 && \
    rm -fr /var/lib/apt/lists/* && \
    rm -rf /etc/apt/sources.list.d/*

RUN npm install --global --quiet npm truffle ganache

FROM base as truffle

RUN mkdir -p /home/app
WORKDIR /home/app

COPY package.json /home/app
COPY package-lock.json /home/app

RUN npm install --quiet

COPY truffle-config.js /home/app
COPY contracts /home/app/contracts
COPY migrations /home/app/migrations/
COPY test /home/app/test/

CMD ["truffle", "version"]

FROM base as ganache

RUN mkdir -p /home
WORKDIR /home
EXPOSE 8545

# ENTRYPOINT ["ganache", "--hostname 0.0.0.0"]
  1. Docker file for celestia
# from https://github.com/rollkit/local-celestia-devnet
FROM ghcr.io/celestiaorg/celestia-app:v1.0.0-rc14 AS celestia-app

FROM ghcr.io/celestiaorg/celestia-node:v0.11.0-rc12

USER root

# hadolint ignore=DL3018
RUN apk --no-cache add \
        curl \
        jq \
    && mkdir /bridge \
    && chown celestia:celestia /bridge

USER celestia

COPY --from=celestia-app /bin/celestia-appd /bin/

COPY entrypoint.sh /opt/entrypoint.sh

EXPOSE 26657 26658 26659 9090

ENTRYPOINT [ "/bin/bash", "/opt/entrypoint.sh" ]
  1. Finally add entrypoint.sh with the following contents
#!/usr/bin/env bash

CHAINID="test"

# App & node has a celestia user with home dir /home/celestia
APP_PATH="/home/celestia/.celestia-app"
NODE_PATH="/home/celestia/bridge/"

#Check if the folder exists
if [ -d "$APP_PATH" ]; then
  # If it exists, do nothing
  echo "Folder exists, using existing config"
else
  # If it doesn't exist, print a message
  echo "The folder $APP_PATH does not exist."
  # Build genesis file incl account for passed address
  coins="1000000000000000utia"
  celestia-appd init $CHAINID --chain-id $CHAINID
  celestia-appd keys add validator --keyring-backend="test"
  # this won't work because some proto types are declared twice and the logs output to stdout (dependency hell involving iavl)
  celestia-appd add-genesis-account $(celestia-appd keys show validator -a --keyring-backend="test") $coins
  celestia-appd gentx validator 5000000000utia \
    --keyring-backend="test" \
    --chain-id $CHAINID

  celestia-appd collect-gentxs

  # Set proper defaults and change ports
  # If you encounter: `sed: -I or -i may not be used with stdin` on MacOS you can mitigate by installing gnu-sed
  # https://gist.github.com/andre3k1/e3a1a7133fded5de5a9ee99c87c6fa0d?permalink_comment_id=3082272#gistcomment-3082272
  sed -i'.bak' 's#"tcp://127.0.0.1:26657"#"tcp://0.0.0.0:26657"#g' ~/.celestia-app/config/config.toml
  sed -i'.bak' 's/timeout_commit = "25s"/timeout_commit = "1s"/g' ~/.celestia-app/config/config.toml
  sed -i'.bak' 's/timeout_propose = "3s"/timeout_propose = "1s"/g' ~/.celestia-app/config/config.toml
  sed -i'.bak' 's/index_all_keys = false/index_all_keys = true/g' ~/.celestia-app/config/config.toml
  sed -i'.bak' 's/mode = "full"/mode = "validator"/g' ~/.celestia-app/config/config.toml

  # Register the validator EVM address
  {
    # wait for block 1
    sleep 20

    # private key: da6ed55cb2894ac2c9c10209c09de8e8b9d109b910338d5bf3d747a7e1fc9eb9
    celestia-appd tx qgb register \
      "$(celestia-appd keys show validator --home "${APP_PATH}" --bech val -a)" \
      0x966e6f22781EF6a6A82BBB4DB3df8E225DfD9488 \
      --from validator \
      --home "${APP_PATH}" \
      --fees 30000utia -b block \
      -y
  } &

  mkdir -p $NODE_PATH/keys
  cp -r $APP_PATH/keyring-test/ $NODE_PATH/keys/keyring-test/

fi

# Start the celestia-app
celestia-appd start &

# Try to get the genesis hash. Usually first request returns an empty string (port is not open, curl fails), later attempts
# returns "null" if block was not yet produced.
GENESIS=
CNT=0
MAX=30
while [ "${#GENESIS}" -le 4 -a $CNT -ne $MAX ]; do
    GENESIS=$(curl -s http://127.0.0.1:26657/block?height=1 | jq '.result.block_id.hash' | tr -d '"')
    ((CNT++))
    sleep 1
done

export CELESTIA_CUSTOM=test:$GENESIS
echo $CELESTIA_CUSTOM

celestia bridge init --node.store /home/celestia/bridge
export CELESTIA_NODE_AUTH_TOKEN=$(celestia bridge auth admin --node.store ${NODE_PATH})
echo "WARNING: Keep this auth token secret **DO NOT** log this auth token outside of development. CELESTIA_NODE_AUTH_TOKEN=$CELESTIA_NODE_AUTH_TOKEN"
celestia bridge start \
  --node.store $NODE_PATH --gateway \
  --core.ip 127.0.0.1 \
  --keyring.accname validator \
  --gateway.deprecated-endpoints

Expected result

Network starts as expected; specifically:

docker-celestia-1  | Folder exists, using existing config
docker-celestia-1  | 2:33PM INF starting node with ABCI Tendermint in-process
docker-celestia-1  | 2:33PM INF service start impl=multiAppConn module=proxy msg={}
docker-celestia-1  | 2:33PM INF service start connection=query impl=localClient module=abci-client msg={}
docker-celestia-1  | 2:33PM INF service start connection=snapshot impl=localClient module=abci-client msg={}
docker-celestia-1  | 2:33PM INF service start connection=mempool impl=localClient module=abci-client msg={}
docker-celestia-1  | 2:33PM INF service start connection=consensus impl=localClient module=abci-client msg={}
docker-celestia-1  | 2:33PM INF service start impl=EventBus module=events msg={}
docker-celestia-1  | 2:33PM INF service start impl=PubSub module=pubsub msg={}
docker-celestia-1  | 2:33PM INF service start impl=IndexerService module=txindex msg={}
docker-celestia-1  | 2:33PM INF ABCI Handshake App Info hash=":����\r�\x05�p#ze2�z%K\x02}v��GF\x0e\x1doe�\x7f�" height=109 module=consensus protocol-version=1 software-version=1.0.0-rc14
docker-celestia-1  | 2:33PM INF ABCI Replay Blocks appHeight=109 module=consensus stateHeight=109 storeHeight=109
docker-celestia-1  | 2:33PM INF Completed ABCI Handshake - CometBFT and App are synced appHash=":����\r�\x05�p#ze2�z%K\x02}v��GF\x0e\x1doe�\x7f�" appHeight=109 module=consensus
docker-celestia-1  | 2:33PM INF Version info abci=0.17.0 block=11 cmtbft_version=0.34.28 commit_hash= p2p=8
docker-celestia-1  | 2:33PM INF This node is a validator addr=BA4385AC51DED3ADB51083D87D0709943AF6391E module=consensus pubKey=bPGawdqQkBd1v+fC8tAAEPo5Mmla+mYkTmUHgEUAhvg=
docker-celestia-1  | 2:33PM INF P2P Node ID ID=a5c248134a7b2552e49326e0da714d88322a3339 file=/home/celestia/.celestia-app/config/node_key.json module=p2p
docker-celestia-1  | 2:33PM INF Adding persistent peers addrs=[] module=p2p
docker-celestia-1  | 2:33PM INF Adding unconditional peer ids ids=[] module=p2p
docker-celestia-1  | 2:33PM INF Add our address to book addr={"id":"a5c248134a7b2552e49326e0da714d88322a3339","ip":"0.0.0.0","port":26656} book=/home/celestia/.celestia-app/config/addrbook.json module=p2p
docker-celestia-1  | 2:33PM INF service start impl=Node msg={}
docker-celestia-1  | 2:33PM INF Starting pprof server laddr=localhost:6060
docker-celestia-1  | 2:33PM INF service start impl="P2P Switch" module=p2p msg={}
docker-celestia-1  | 2:33PM INF service start impl=Mempool msg={}
docker-celestia-1  | 2:33PM INF serve module=rpc-server msg={}
docker-celestia-1  | 2:33PM INF service start impl=BlockchainReactor module=blockchain msg={}
docker-celestia-1  | 2:33PM INF service start impl=ConsensusReactor module=consensus msg={}
docker-celestia-1  | 2:33PM INF Reactor  module=consensus waitSync=false
docker-celestia-1  | 2:33PM INF service start impl=ConsensusState module=consensus msg={}
docker-celestia-1  | 2:33PM INF service start impl=baseWAL module=consensus msg={} wal=/home/celestia/.celestia-app/data/cs.wal/wal
docker-celestia-1  | 2:33PM INF service start impl=Group module=consensus msg={} wal=/home/celestia/.celestia-app/data/cs.wal/wal
docker-celestia-1  | 2:33PM INF service start impl=TimeoutTicker module=consensus msg={}
docker-celestia-1  | 2:33PM INF Searching for height height=110 max=0 min=0 module=consensus wal=/home/celestia/.celestia-app/data/cs.wal/wal
docker-celestia-1  | 2:33PM INF Searching for height height=109 max=0 min=0 module=consensus wal=/home/celestia/.celestia-app/data/cs.wal/wal
docker-celestia-1  | 2:33PM INF Found height=109 index=0 module=consensus wal=/home/celestia/.celestia-app/data/cs.wal/wal
docker-celestia-1  | 2:33PM INF Catchup by replaying consensus messages height=110 module=consensus
docker-celestia-1  | 2:33PM INF Replay: Done module=consensus
docker-celestia-1  | 2:33PM INF service start impl=Evidence module=evidence msg={}
docker-celestia-1  | 2:33PM INF service start impl=StateSync module=statesync msg={}
docker-celestia-1  | 2:33PM INF service start impl=PEX module=pex msg={}
docker-celestia-1  | 2:33PM INF service start book=/home/celestia/.celestia-app/config/addrbook.json impl=AddrBook module=p2p msg={}
docker-celestia-1  | 2:33PM INF Saving AddrBook to file book=/home/celestia/.celestia-app/config/addrbook.json module=p2p size=0
docker-celestia-1  | 2:33PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
docker-celestia-1  | 2:33PM INF No addresses to dial. Falling back to seeds module=pex
docker-celestia-1  | 2:33PM INF starting API server... module=api-server
docker-celestia-1  | 2:33PM INF serve module=api-server msg={}

Actual result

Celestia light client fails to start meaning that the following output is produced instead of the expected.

Relevant log output

Attaching to docker-celestia-1, docker-eth-1
docker-celestia-1  | The folder /home/celestia/.celestia-app does not exist.
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: accepts 2 arg(s), received 1
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | mkdir: can't create directory '/home/celestia/bridge//': Permission denied
docker-celestia-1  | cp: can't stat '/home/celestia/.celestia-app/keyring-test/': No such file or directory
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied

Notes

Permissions for the actual location have been checked with chmod 777 to ensure that the permissions were actually set right. It's possible that the location isn't exactly what we expect and the mistake is in the docker-compose.yml.

This problem can be worked around if in the docker compose file the bind mount is disabled: 

# volumes:
    # - ./.local_data/celestia:/home/celestia
Bidon15 commented 1 year ago

Dear @appetrosyan,

Thank you so much for opening this bug. After some debugging, I've found some nits that need to be adapted in order to make the compose.yaml run as expected. Please take a look at this yaml snippet: 

version: '3'
services:
  celestia:
    user: 10001:10001
    image: ghcr.io/rollkit/local-celestia-devnet:v0.11.0-rc12
    volumes:
      - ./.local_data/celestia:/home/celestia
    ports:
      - "26657:26657"
      - "26658:26658"
      - "26659:26659"
      - "9090:9090"

  #use ganache for now
  eth:
    image: trufflesuite/ganache:latest
    ports:
      - "8545:8545"
    volumes:
      - ./.local_data/ganache_data:/ganache_data
    command:
      - --database.dbPath=/ganache_data
      - -m
      - 'minimum symptom minute gloom tragic situate silver mechanic salad amused elite beef'
      - -h
      - '0.0.0.0'

As you can see, we have missed the user group assignment in docker-compose. After we've provided the user: 10001:10001(which is equal to user: celestia:celestia), the celestia app/node part is going to be started as expected.

I hope you are fine that ganache part has been redacted to make sure you can have expected behaviour of eth local blockchain. ganache-cli has been ditched in favour of ganache image. In addition, the image is using commands as args flags etc, so we have command instead of entrypoint.

Let me know if it looks fine for you.

Here is proof that this solution works in my mac: https://app.warp.dev/block/pLcBHNpAOdPlEA34iRlWkv

Cheers

appetrosyan commented 1 year ago

It didn't exactly solve my problem. 

app@antares ~/G/s/docker (main)> docker compose --file docker-compose-revised.yml  up --force-recreate 
[+] Running 2/2
 ✔ Container docker-eth-1       Recreated                                                                                                                                  0.1s 
 ✔ Container docker-celestia-1  Recreated                                                                                                                                  0.1s 
Attaching to docker-celestia-1, docker-eth-1
docker-celestia-1  | The folder /home/celestia/.celestia-app does not exist.
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: accepts 2 arg(s), received 1
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | sed: /home/celestia/.celestia-app/config/config.toml: No such file or directory
docker-celestia-1  | mkdir: can't create directory '/home/celestia/bridge//': Permission denied
docker-celestia-1  | cp: can't stat '/home/celestia/.celestia-app/keyring-test/': No such file or directory
docker-celestia-1  | Error: couldn't make client config: mkdir /home/celestia/.celestia-app: permission denied

I think it's probably something on our end, but it'd be nice if your experts could help us track down the problem.

ramin commented 1 year ago

@appetrosyan will you make a public repo with exact setup of the current docker-compose.yml? I'll pull it down and try and re-create too then work through it with you

appetrosyan commented 1 year ago

I found the culprit

ramin commented 1 year ago

@appetrosyan amazing, let us know what it was

appetrosyan commented 1 year ago

So we're still not quite operational yet (it's a different error message), but here's what it was. The files created by entrypoint.sh were created as root, with 0600 permissions. So it wasn't even possible to figure out if the directory existed.

Weirdly, only setting it to chmod 707 (it's really weird, but I checked other less permissive combinations), allowed the process to get past the initial stage.

Now the program terminates with a segfault using this Dockerfile

# from https://github.com/rollkit/local-celestia-devnet
FROM ghcr.io/celestiaorg/celestia-app:v1.0.0-rc14 AS celestia-app

FROM ghcr.io/celestiaorg/celestia-node:v0.11.0-rc12

USER root

# hadolint ignore=DL3018
RUN apk --no-cache add \
        curl \
        jq \
    && mkdir /bridge \
    && chown celestia:celestia /bridge

USER celestia

COPY --from=celestia-app /bin/celestia-appd /bin/

COPY entrypoint.sh /opt/entrypoint.sh

EXPOSE 26657 26658 26659 9090

ENTRYPOINT [ "/bin/bash", "/opt/entrypoint.sh" ]
Segfault logs ``` docker-celestia-1 | Folder exists, using existing config docker-celestia-1 | 2:07PM INF starting node with ABCI Tendermint in-process docker-celestia-1 | 2:07PM INF service start impl=multiAppConn module=proxy msg={} docker-celestia-1 | 2:07PM INF service start connection=query impl=localClient module=abci-client msg={} docker-celestia-1 | 2:07PM INF service start connection=snapshot impl=localClient module=abci-client msg={} docker-celestia-1 | 2:07PM INF service start connection=mempool impl=localClient module=abci-client msg={} docker-celestia-1 | 2:07PM INF service start connection=consensus impl=localClient module=abci-client msg={} docker-celestia-1 | 2:07PM INF service start impl=EventBus module=events msg={} docker-celestia-1 | 2:07PM INF service start impl=PubSub module=pubsub msg={} docker-celestia-1 | 2:07PM INF service start impl=IndexerService module=txindex msg={} docker-celestia-1 | 2:07PM INF ABCI Handshake App Info hash="\x1e���P�����.{��]P\n����:�~\a�g��0��" height=2 module=consensus protocol-version=1 software-version=1.0.0-rc14 docker-celestia-1 | 2:07PM INF ABCI Replay Blocks appHeight=2 module=consensus stateHeight=2 storeHeight=2 docker-celestia-1 | 2:07PM INF Completed ABCI Handshake - CometBFT and App are synced appHash="\x1e���P�����.{��]P\n����:�~\a�g��0��" appHeight=2 module=consensus docker-celestia-1 | 2:07PM INF Version info abci=0.17.0 block=11 cmtbft_version=0.34.28 commit_hash= p2p=8 docker-celestia-1 | 2:07PM INF This node is a validator addr=59BB96DAA96CD34938011AFC557BD35C54E6608C module=consensus pubKey=Zh/im7lpWHFm8r+CT+TsfEuRJyUF7QISVPm1jr41N5M= docker-celestia-1 | 2:07PM INF P2P Node ID ID=ee4677c3be14aee48754da7aa5251092bff001c5 file=/home/celestia/.celestia-app/config/node_key.json module=p2p docker-celestia-1 | 2:07PM INF Adding persistent peers addrs=[] module=p2p docker-celestia-1 | 2:07PM INF Adding unconditional peer ids ids=[] module=p2p docker-celestia-1 | 2:07PM INF Add our address to book addr={"id":"ee4677c3be14aee48754da7aa5251092bff001c5","ip":"0.0.0.0","port":26656} book=/home/celestia/.celestia-app/config/addrbook.json module=p2p docker-celestia-1 | 2:07PM INF service start impl=Node msg={} docker-celestia-1 | 2:07PM INF Starting pprof server laddr=localhost:6060 docker-celestia-1 | 2:07PM INF service start impl="P2P Switch" module=p2p msg={} docker-celestia-1 | 2:07PM INF service start impl=Evidence module=evidence msg={} docker-celestia-1 | 2:07PM INF serve module=rpc-server msg={} docker-celestia-1 | 2:07PM INF service start impl=StateSync module=statesync msg={} docker-celestia-1 | 2:07PM INF service start impl=PEX module=pex msg={} docker-celestia-1 | 2:07PM INF service start book=/home/celestia/.celestia-app/config/addrbook.json impl=AddrBook module=p2p msg={} docker-celestia-1 | 2:07PM INF service start impl=Mempool msg={} docker-celestia-1 | 2:07PM INF service start impl=BlockchainReactor module=blockchain msg={} docker-celestia-1 | 2:07PM INF service start impl=ConsensusReactor module=consensus msg={} docker-celestia-1 | 2:07PM INF Reactor module=consensus waitSync=false docker-celestia-1 | 2:07PM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10 docker-celestia-1 | 2:07PM INF No addresses to dial. Falling back to seeds module=pex docker-celestia-1 | 2:07PM INF service start impl=ConsensusState module=consensus msg={} docker-celestia-1 | 2:07PM INF service start impl=baseWAL module=consensus msg={} wal=/home/celestia/.celestia-app/data/cs.wal/wal docker-celestia-1 | 2:07PM INF service start impl=Group module=consensus msg={} wal=/home/celestia/.celestia-app/data/cs.wal/wal docker-celestia-1 | 2:07PM INF service start impl=TimeoutTicker module=consensus msg={} docker-celestia-1 | 2:07PM INF Searching for height height=3 max=0 min=0 module=consensus wal=/home/celestia/.celestia-app/data/cs.wal/wal docker-celestia-1 | 2:07PM INF Searching for height height=2 max=0 min=0 module=consensus wal=/home/celestia/.celestia-app/data/cs.wal/wal docker-celestia-1 | 2:07PM INF Found height=2 index=0 module=consensus wal=/home/celestia/.celestia-app/data/cs.wal/wal docker-celestia-1 | 2:07PM INF Catchup by replaying consensus messages height=3 module=consensus docker-celestia-1 | 2:07PM INF Replay: New Step height=3 module=consensus round=0 step=RoundStepNewHeight docker-celestia-1 | 2:07PM INF Replay: Done module=consensus docker-celestia-1 | 2:07PM INF Saving AddrBook to file book=/home/celestia/.celestia-app/config/addrbook.json module=p2p size=0 docker-celestia-1 | 2:07PM INF starting API server... module=api-server docker-celestia-1 | 2:07PM INF serve module=api-server msg={} docker-celestia-1 | test:AFA16514AC3D59FC6A647A61A9743498EE3263ED00195B61239EF8165296F07B docker-celestia-1 | docker-celestia-1 | docker-celestia-1 | WARNING: Celestia custom network specified. Only use this option if the node is freshly created and initialized. docker-celestia-1 | **DO NOT** run a custom network over an already-existing node store! docker-celestia-1 | docker-celestia-1 | docker-celestia-1 | 2023-10-04T14:07:26.487Z INFO node nodebuilder/init.go:31 Initializing Bridge Node Store over '/home/celestia/bridge' docker-celestia-1 | 2023-10-04T14:07:26.487Z INFO node nodebuilder/init.go:63 Saved config {"path": "/home/celestia/bridge/config.toml"} docker-celestia-1 | 2023-10-04T14:07:26.487Z INFO node nodebuilder/init.go:65 Accessing keyring... docker-celestia-1 | 2023-10-04T14:07:26.491Z WARN node nodebuilder/init.go:194 Detected plaintext keyring backend. For elevated security properties, consider using the `file` keyring backend. docker-celestia-1 | panic: runtime error: invalid memory address or nil pointer dereference docker-celestia-1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xe2b9ee] docker-celestia-1 | docker-celestia-1 | goroutine 1 [running]: docker-celestia-1 | github.com/99designs/keyring.(*fileKeyring).resolveDir(0x2246962?) docker-celestia-1 | /go/pkg/mod/github.com/99designs/keyring@v1.2.1/file.go:48 +0x8e docker-celestia-1 | github.com/99designs/keyring.(*fileKeyring).Keys(0x0?) docker-celestia-1 | /go/pkg/mod/github.com/99designs/keyring@v1.2.1/file.go:169 +0x17 docker-celestia-1 | github.com/cosmos/cosmos-sdk/crypto/keyring.keystore.MigrateAll({{0x36a71e0, 0xc00160def0}, {0x36c4c20, 0xc001687be0}, {0x29547b6, 0x4}, {{0xc001687c60, 0x1, 0x1}, {0xc001687c70, ...}}}) docker-celestia-1 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.0-sdk-v0.46.14/crypto/keyring/keyring.go:838 +0x30 docker-celestia-1 | github.com/cosmos/cosmos-sdk/crypto/keyring.keystore.List(...) docker-celestia-1 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.0-sdk-v0.46.14/crypto/keyring/keyring.go:495 docker-celestia-1 | github.com/celestiaorg/celestia-node/nodebuilder.generateKeys({{0x1bf08eb000, 0x1bf08eb000}, { docker-celestia-1 | {0x0, 0x0} docker-celestia-1 | , {0xc00097d1b0, 0x5}, {0xc00097d1b8, 0x4}}, {{0x0, 0x0}, ...}, ...}, ...) docker-celestia-1 | /src/nodebuilder/init.go:201 +0x1c5 docker-celestia-1 | github.com/celestiaorg/celestia-node/nodebuilder.Init({{ docker-celestia-1 | 0x1bf08eb000, 0x1bf08eb000}, {{0x0, 0x0}, docker-celestia-1 | {0xc00097d1b0, 0x5}, {0xc00097d1b8, 0x4}}, {{0x0, 0x0}, ...}, ...}, ...) docker-celestia-1 | /src/nodebuilder/init.go:66 +0x588 docker-celestia-1 | github.com/celestiaorg/celestia-node/cmd.Init.func1( docker-celestia-1 | 0xc000629200?, {0xc0009b82c0 docker-celestia-1 | ?, 0x4?, docker-celestia-1 | 0x29546de?}) docker-celestia-1 | /src/cmd/init.go: docker-celestia-1 | 19 +0x128 docker-celestia-1 | github.com/spf13/cobra.(*Command).execute(0xc000532900, {0xc0009b82a0, 0x2, 0x2} docker-celestia-1 | ) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c docker-celestia-1 | github.com/spf13/cobra.(*Command).ExecuteC(0x4d45040) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 docker-celestia-1 | github.com/spf13/cobra.(*Command).Execute(...) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 docker-celestia-1 | github.com/spf13/cobra.(*Command).ExecuteContext( docker-celestia-1 | ...) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:985 docker-celestia-1 | main.run() docker-celestia-1 | /src/cmd/celestia/main.go:28 docker-celestia-1 | +0x4f docker-celestia-1 | main.main() docker-celestia-1 | /src/cmd/celestia/main.go:21 + docker-celestia-1 | 0x13 docker-celestia-1 | Error: keystore: check before reading key 'jwt-secret.jwt' failed: stat /home/celestia/bridge/keys/NJ3XILLTMVRXEZLUFZVHO5A: permission denied docker-celestia-1 | Usage: docker-celestia-1 | celestia bridge auth [permission-level (e.g. read || write || admin)] [flags] docker-celestia-1 | docker-celestia-1 | Flags: docker-celestia-1 | --core.grpc.port string Set a custom gRPC port for the core node connection. The --core.ip flag must also be provided. (default "9090") docker-celestia-1 | --core.ip string Indicates node to connect to the given core node. Example: , 127.0.0.1. , subdomain.domain.tld Assumes RPC port 26657 and gRPC port 9090 as default unless otherwise specified. docker-celestia-1 | --core.rpc.port string Set a custom RPC port for the core node connection. The --core.ip flag must also be provided. (default "26657") docker-celestia-1 | --gateway Enables the REST gateway docker-celestia-1 | --gateway.addr string Set a custom gateway listen address (default: localhost) docker-celestia-1 | --gateway.deprecated-endpoints Enables deprecated endpoints on the gateway. These will be removed in the next release. docker-celestia-1 | --gateway.port string Set a custom gateway port (default: 26659) docker-celestia-1 | -h, --help help for auth docker-celestia-1 | --keyring.accname string Directs node's keyring signer to use the key prefixed with the given string. docker-celestia-1 | --keyring.backend string Directs node's keyring signer to use the given backend. Default is test. (default "test") docker-celestia-1 | --log.level string DEBUG, INFO, WARN, ERROR, DPANIC, PANIC, FATAL docker-celestia-1 | and their lower-case forms (default "INFO") docker-celestia-1 | --log.level.module strings :, e.g. pubsub:debug docker-celestia-1 | --metrics Enables OTLP metrics with HTTP exporter docker-celestia-1 | --metrics.endpoint string Sets HTTP endpoint for OTLP metrics to be exported to. Depends on '--metrics' (default "localhost:4318") docker-celestia-1 | --metrics.tls Enable TLS connection to OTLP metric backend (default true) docker-celestia-1 | --node.config string Path to a customized node config TOML file docker-celestia-1 | --node.store string The path to root/home directory of your Celestia Node Store docker-celestia-1 | --p2p.metrics Enable libp2p metrics docker-celestia-1 | --p2p.mutual strings Comma-separated multiaddresses of mutual peers to keep a prioritized connection with. docker-celestia-1 | Such connection is immune to peer scoring slashing and connection module trimming. docker-celestia-1 | Peers must bidirectionally point to each other. (Format: multiformats.io/multiaddr) docker-celestia-1 | docker-celestia-1 | --p2p.network string The name of the network to connect to, e.g. arabica-10, mocha-4, blockspacerace-0. Must be passed on both init and start to take effect. docker-celestia-1 | --pprof Enables standard profiling handler (pprof) and exposes the profiles on port 6000 docker-celestia-1 | --pyroscope Enables Pyroscope profiling docker-celestia-1 | --pyroscope.endpoint string Sets HTTP endpoint for Pyroscope profiles to be exported to. Depends on '--pyroscope' (default "http://localhost:4040") docker-celestia-1 | --pyroscope.tracing Enables Pyroscope tracing integration. Depends on --tracing docker-celestia-1 | --rpc.addr string Set a custom RPC listen address (default: localhost) docker-celestia-1 | --rpc.port string Set a custom RPC port (default: 26658) docker-celestia-1 | --tracing Enables OTLP tracing with HTTP exporter docker-celestia-1 | --tracing.endpoint string Sets HTTP endpoint for OTLP traces to be exported to. Depends on '--tracing' (default "localhost:4318") docker-celestia-1 | --tracing.tls Enable TLS connection to OTLP tracing backend (default true) docker-celestia-1 | docker-celestia-1 | docker-celestia-1 | WARNING: Keep this auth token secret **DO NOT** log this auth token outside of development. CELESTIA_NODE_AUTH_TOKEN= docker-celestia-1 | docker-celestia-1 | WARNING: Celestia custom network specified. Only use this option if the node is freshly created and initialized. docker-celestia-1 | **DO NOT** run a custom network over an already-existing node store! docker-celestia-1 | WARNING: Keep this auth token secret **DO NOT** log this auth token outside of development. CELESTIA_NODE_AUTH_TOKEN= docker-celestia-1 | docker-celestia-1 | WARNING: Celestia custom network specified. Only use this option if the node is freshly created and initialized. docker-celestia-1 | **DO NOT** run a custom network over an already-existing node store! docker-celestia-1 | docker-celestia-1 | docker-celestia-1 | docker-celestia-1 | WARNING: Celestia custom network specified. Only use this option if the node is freshly created and initialized. docker-celestia-1 | **DO NOT** run a custom network over an already-existing node store! docker-celestia-1 | docker-celestia-1 | docker-celestia-1 | 2023-10-04T14:07:26.601Z INFO node nodebuilder/module.go:24 Accessing keyring... docker-celestia-1 | panic: runtime error: invalid memory address or nil pointer dereference docker-celestia-1 | [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xe2b9ee] docker-celestia-1 | docker-celestia-1 | goroutine 1 [running]: docker-celestia-1 | github.com/99designs/keyring.(*fileKeyring).resolveDir(0x100c00159f060?) docker-celestia-1 | /go/pkg/mod/github.com/99designs/keyring@v1.2.1/file.go:48 +0x8e docker-celestia-1 | github.com/99designs/keyring.(*fileKeyring).filename(0x28e7800?, {0xc0014f8400 docker-celestia-1 | , 0xe}) docker-celestia-1 | /go/pkg/mod/github.com/99designs/keyring@v1.2.1/file.go:151 + docker-celestia-1 | 0x25 docker-celestia-1 | github.com/99designs/keyring.(*fileKeyring).Get(0xc001519650, {0xc0014f8400?, 0x50fb480?}) docker-celestia-1 | docker-celestia-1 | /go/pkg/mod/github.com/99designs/keyring@v1.2.1/file.go:73 +0x45 docker-celestia-1 | github.com/cosmos/cosmos-sdk/crypto/keyring.keystore.migrate({{0x36a71e0, 0xc001519650} docker-celestia-1 | , {0x36c4c20, 0xc0014ebd10}, {0x29547b6, 0x4}, {{0xc0014ebd40, 0x1, 0x1}, {0xc0014ebd50 docker-celestia-1 | , ...}}}, ...) docker-celestia-1 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.0-sdk-v0.46.14/crypto/keyring/keyring.go:885 +0x132 docker-celestia-1 | github.com/cosmos/cosmos-sdk/crypto/keyring.keystore.Key(...) docker-celestia-1 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.0-sdk-v0.46.14/crypto/keyring/keyring.go:559 docker-celestia-1 | github.com/celestiaorg/celestia-node/nodebuilder/state.KeyringSigner({{0x7ffd4492cdd0?, 0x0?}, {0x29547b6?, 0xc000825c20?}}, {0x36ab148, 0xc001545d60}, {0xc000058010, 0x4}) docker-celestia-1 | /src/nodebuilder/state/keyring.go:22 +0x9f docker-celestia-1 | github.com/celestiaorg/celestia-node/nodebuilder.ConstructModule(0x1, {0xc000058010, 0x4}, 0xc0002db340, {0x36ab190, 0xc000510c80}) docker-celestia-1 | /src/nodebuilder/module.go:29 +0xde docker-celestia-1 | github.com/celestiaorg/celestia-node/nodebuilder.NewWithConfig docker-celestia-1 | (0x89?, {0xc000058010?, docker-celestia-1 | 0x50f8500?}, { docker-celestia-1 | 0x36ab190?, 0xc000510c80?}, 0x29884a2? docker-celestia-1 | , {0xc00159fba0, 0x0, 0x4?}) docker-celestia-1 | /src/nodebuilder/node.go: docker-celestia-1 | 90 +0x4b docker-celestia-1 | github.com/celestiaorg/celestia-node/cmd.Start.func1(0xc000a40000, {0xc000746b00?, 0x4? docker-celestia-1 | , 0x29546de?}) docker-celestia-1 | /src/cmd/start.go:55 +0x465 docker-celestia-1 | github.com/spf13/cobra.(*Command).execute(0xc000a40000, {0xc000746a80, 0x8, 0x8}) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:940 +0x87c docker-celestia-1 | github.com/spf13/cobra.(*Command).ExecuteC(0x4d45040) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:1068 +0x3a5 docker-celestia-1 | github.com/spf13/cobra.(*Command).Execute(...) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go:992 docker-celestia-1 | github.com/spf13/cobra.(*Command).ExecuteContext(...) docker-celestia-1 | /go/pkg/mod/github.com/spf13/cobra@v1.7.0/command.go docker-celestia-1 | :985 docker-celestia-1 | main.run() docker-celestia-1 | /src/cmd/celestia/main.go:28 + docker-celestia-1 | 0x4f docker-celestia-1 | main.main docker-celestia-1 | () docker-celestia-1 | /src/cmd/celestia/main.go:21 +0x13 ```

Comment

I'm really surprised and a little baffled by that both the owner and everyone else require the execution right. This might be related to the segfault, maybe something gets executed, instead of being interpreted as plain data?

appetrosyan commented 1 year ago

However, if we use the ghcr.io/rollkit/local-celestia-devnet:v0.11.0-rc12 image, everything works fine, so I'm now trying to work out what our image is doing differently.

ramin commented 1 year ago

great! sounds like you are on the path to fixing here, i'm going to close out the issue.