Monerod Container starting but never running

ubuntophe commented 1 month ago

After a new install of monerod and running docker compose up -d monerod I get this Container monerod Started But after checking the status docker exec monerod /usr/local/bin/monerod status I keep having the same message: Error response from daemon: Container 3d883896f79fad842eeb27c18b30536c1f0e94ea87b7d169a46b820cdceae5e7 is restarting, wait until the container is running

I tried reboot, I left it sleep for a night, but still same result.

I run the container on Ubuntu 24.04 on a RPi5.

ubuntophe commented 1 month ago

From docker logs, this seems to be the problem :

I Initializing core...
I Loading blockchain from folder /home/monero/.bitmonero/lmdb ...
F Existing lmdb database is incompatible with this version.
F Please delete the existing database and resync.
I Stopping cryptonote protocol...
I Cryptonote protocol stopped successfully
E Exception in main! Failed to initialize core

hundehausen commented 1 month ago

Hey @ubuntophe,

can you share your docker-compse.yml file here? So I can look at what is different from Monero Suite, so I can find out, what the pronlem was.

ubuntophe commented 1 month ago

I changed completely the docker-compse.yml file and don't have a copy of the one that was causing the initial problem. I now use a mix of Monero Suite and Seth repo. Not sure it is a good idea though. It seemed to work ok as docker exec monerod /usr/local/bin/monerod status showed it was initializing.

Now I can't ssh into the session anymore. I get this message ssh_exchange_identification: read: Connection reset by peer No sure what to do.

Anyway, this is the mixed repo I use:

version: '3.5'
services:
  monerod:
    image: ghcr.io/sethforprivacy/simple-monerod:latest
    restart: unless-stopped
    container_name: monerod
    volumes:
      - bitmonero:/home/monero/.bitmonero
    ports:
      - 18080:18080
      - 18084:18084
      - 18089:18089
    command:
      - --rpc-restricted-bind-ip=0.0.0.0
      - --rpc-restricted-bind-port=18089
      - --rpc-bind-ip=0.0.0.0
      - --rpc-bind-port=18081
      - --confirm-external-bind
      - --enable-dns-blocklist
      - --check-updates=disabled
      - --max-log-files=3
      - --max-log-file-size=1048576
      - --no-igd
      - --out-peers=64
      - --limit-rate-down=1048576
      - --public-node
      - --zmq-pub=tcp://0.0.0.0:18084
  p2pool:
    image: ghcr.io/sethforprivacy/p2pool:latest
    restart: unless-stopped
    container_name: p2pool
    tty: true
    stdin_open: true
    volumes:
      - p2pool-data:/home/p2pool
      - /dev/hugepages:/dev/hugepages:rw
    ports:
      - 3333:3333
      - 37888:37888
    command: >-
      --wallet "MY WALLET ADDRESS"
    --stratum "0.0.0.0:3333" --p2p "0.0.0.0:37889" --zmq-port "18084"
      --loglevel "0" --addpeers "65.21.227.114:37889,node.sethforprivacy.com:37889"
      --host "monerod" --rpc-port "18089" --mini 
monero-wallet-rpc:
    image: sethsimmons/simple-monero-wallet-rpc:latest
    restart: unless-stopped
    container_name: monero-wallet-rpc
    ports:
      - 127.0.0.1:18083:18083
    volumes:
      - monero-wallet-rpc-data:/home/monero
    command:
      - --daemon-address=monerod:18089
      - --trusted-daemon
      - --rpc-bind-port=18083
      - --wallet-dir /home/monero
  moneroblock:
    image: sethsimmons/moneroblock:latest
    restart: unless-stopped
    container_name: moneroblock
    ports:
      - 127.0.0.1:31312:31312
    command:
      - --daemon
      - monerod:18089

watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    environment:
      WATCHTOWER_CLEANUP: true
      WATCHTOWER_POLL_INTERVAL: 3600
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

autoheal:
    image: willfarrell/autoheal:latest
    container_name: autoheal
    restart: unless-stopped
    environment:
      AUTOHEAL_CONTAINER_LABEL: all
    volumes:
- /var/run/docker.sock:/var/run/docker.sock

volumes:
  bitmonero: {}
  p2pool-data: {}
  monero-wallet-rpc-data: {}

ubuntophe commented 1 month ago

I tried using Seth repo without any changes and I get this from docker logs

Initializing core...
I Loading blockchain from folder /home/monero/.bitmonero/lmdb ...
W Attempt to get timestamp from height 85667 failed -- timestamp not in db
I Stopping cryptonote protocol...
I Cryptonote protocol stopped successfully
E Exception in main! Attempt to get timestamp from height 85667 failed -- timestamp not in db

What can be done?

dearkafka commented 1 month ago

same issue for me, I feel like autoheal is killing monerod earlier than it can initialize db. (hypothesis)

dearkafka commented 1 month ago

there was a change by @sethforprivacy sethforprivacy/simple-monerod-docker@3a7ec6a and I feel like it's not enough to be responsive for this method.

I did in compose for monerod like this, increasing default retries and interval, seems to be ok.

    healthcheck:
      test: ["CMD-SHELL", "curl --fail http://localhost:18081/get_height || exit 1"]
      interval: 60s
      timeout: 5s
      retries: 10
      start_period: 40s

ubuntophe commented 2 days ago

It's me again. I tried again to get Monero node working. But it got stuck again in the initial synchronization.

I used the exact docker-compose.yml from monero-suite, without any changes.

It synched normal for a few hours then the logs show this :

2024-06-22 21:28:11.094 I Synced 1351116/3177194 (42%, 1826078 left)
2024-06-22 21:28:34.652 I Host 65.109.235.207 blocked.
2024-06-22 21:29:22.675 I Host 147.135.136.35 blocked.
2024-06-22 21:29:46.688 I Host 100.42.27.231 blocked.
2024-06-22 21:30:26.710 I Host 103.57.251.200 blocked.
2024-06-22 21:30:58.729 I Host 170.205.30.38 blocked.
2024-06-22 21:31:46.771 I [76.156.71.240:18080 OUT] Sync data returned a new top block candidate: 1351116 -> 3177195 [Your node is 1826079 blocks (6.9 years) behind] 
2024-06-22 21:31:46.771 I SYNCHRONIZATION started
2024-06-22 21:31:54.758 I Host 34.122.183.94 blocked.
2024-06-22 21:32:02.763 I Host 144.217.11.233 blocked.
2024-06-22 21:32:34.783 I Host 188.243.210.128 blocked.
2024-06-22 21:32:42.786 I Host 76.156.71.240 blocked.
2024-06-22 21:33:06.798 I Host 205.178.124.103 blocked.

Hosts kept being blocked. Then I could not get logs any more:

error from daemon in stream: Error grabbing logs: invalid character '{' after object key:value pair

After reboot, monerod container (+p2pool and rpc) get stuck restarting.

docker logs --tail 40 2ea8afccd3c4 gives this indefinitely :

2024-06-23 18:49:09.369 I Monero 'Fluorine Fermi' (v0.18.3.3-release)
2024-06-23 18:49:09.369 I Initializing cryptonote protocol...
2024-06-23 18:49:09.369 I Cryptonote protocol initialized OK
2024-06-23 18:49:09.370 I Initializing core...
2024-06-23 18:49:09.370 I Loading blockchain from folder /home/monero/.bitmonero/lmdb ...

What can I do?

dearkafka commented 2 days ago

@ubuntophe ok, so "blocked" is normal, it's not related. not 100% sure about your error, but it might be that blockchain db became corrupted. I have never experienced this but I know it happens. Also just out of curiosity you can try changing log level for monerod to see more. for example adding "--log-level=2" but be careful as depending on it there could be too much disk space consumption but you can add something like this "--max-log-files=2" to limit.

ubuntophe commented 2 days ago

docker --log-level=2 --max-log-files=2 2ea8afccd3c4 gives Unable to parse logging level: 2

dearkafka commented 2 days ago

@ubuntophe ok try deleting all log files and restarting maybe?

ubuntophe commented 2 days ago

No luck, I deleted all log files and restart monerod and still the same Unable to parse logging level: 2

dearkafka commented 2 days ago

maybe some extra space or indent? mine works fine:

    command:
      - "--rpc-restricted-bind-ip=0.0.0.0"
      - "--rpc-restricted-bind-port=18089"
      - "--no-igd"
      - "--no-zmq"
      - "--enable-dns-blocklist"
      - "--prune-blockchain"
      - "--log-level=2"
      - "--max-log-files=2"

ubuntophe commented 2 days ago

Can you give me more detailed instructions on what to input? I'm not super proficient.

dearkafka commented 2 days ago

I assume you have this (shared earlier)

version: '3.5'
services:
  monerod:
    image: ghcr.io/sethforprivacy/simple-monerod:latest
    restart: unless-stopped
    container_name: monerod
    volumes:
      - bitmonero:/home/monero/.bitmonero
    ports:
      - 18080:18080
      - 18084:18084
      - 18089:18089
    command:
      - --rpc-restricted-bind-ip=0.0.0.0
      - --rpc-restricted-bind-port=18089
      - --rpc-bind-ip=0.0.0.0
      - --rpc-bind-port=18081
      - --confirm-external-bind
      - --enable-dns-blocklist
      - --check-updates=disabled
      - --max-log-files=3
      - --max-log-file-size=1048576
      - --no-igd
      - --out-peers=64
      - --limit-rate-down=1048576
      - --public-node
      - --zmq-pub=tcp://0.0.0.0:18084

try smth like

version: '3.5'
services:
  monerod:
    image: ghcr.io/sethforprivacy/simple-monerod:latest
    restart: unless-stopped
    container_name: monerod
    volumes:
      - bitmonero:/home/monero/.bitmonero
    ports:
      - 18080:18080
      - 18084:18084
      - 18089:18089
    command:
      - "--rpc-restricted-bind-ip=0.0.0.0"
      - "--rpc-restricted-bind-port=18089"
      - "--rpc-bind-ip=0.0.0.0"
      - "--rpc-bind-port=18081"
      - "--confirm-external-bind"
      - "--enable-dns-blocklist"
      - "--check-updates=disabled"
      - "--max-log-files=3"
      - "--max-log-file-size=1048576"
      - "--no-igd"
      - "--out-peers=64"
      - "--limit-rate-down=1048576"
      - "--public-node"
      - "--zmq-pub=tcp://0.0.0.0:18084"
      - "--log-level=2"

ubuntophe commented 2 days ago

I've added the line to docker-compose.yml and restart, but docker --log-level=2 --max-log-files=2 2ea8afccd3c4 still gives Unable to parse logging level: 2

hundehausen commented 2 days ago

@ubuntophe can you share your whole docker-compose.yml file?

ubuntophe commented 2 days ago

name: monero-suite
services:
  monerod:
    image: ghcr.io/sethforprivacy/simple-monerod:latest
    restart: unless-stopped
    container_name: monerod
    volumes:
      - bitmonero:/home/monero/.bitmonero
    ports:
      - 18080:18080
      - 18084:18084
      - 18089:18089
    healthcheck:
      test: curl --fail http://localhost:18081/get_height || exit 1
      interval: 60s
      timeout: 5s
      retries: 10
      start_period: 40s
    command:
      - --rpc-restricted-bind-ip=0.0.0.0
      - --rpc-restricted-bind-port=18089
      - --rpc-bind-ip=0.0.0.0
      - --rpc-bind-port=18081
      - --confirm-external-bind
      - --enable-dns-blocklist
      - --check-updates=disabled
      - --max-log-files=3
      - --max-log-file-size=1048576
      - --no-igd
      - --out-peers=64
      - --limit-rate-down=1048576
      - --public-node
      - --zmq-pub=tcp://0.0.0.0:18084
      - --log-level=2
      - --max-log-files=2
  p2pool:
    image: ghcr.io/sethforprivacy/p2pool:latest
    restart: unless-stopped
    container_name: p2pool
    tty: true
    stdin_open: true
    volumes:
      - p2pool-data:/home/p2pool
      - /dev/hugepages:/dev/hugepages:rw
    ports:
      - 3333:3333
      - 37888:37888
    command: --wallet
      MYWALLETADDRESS
      --stratum 0.0.0.0:3333 --p2p 0.0.0.0:37888 --rpc-port 18089 --zmq-port
      18084 --host monerod --addpeers node.portemonero.com:37888 --mini
      --start-mining 2
  monero-wallet-rpc:
    image: sethsimmons/simple-monero-wallet-rpc:latest
    restart: unless-stopped
    container_name: monero-wallet-rpc
    ports:
      - 127.0.0.1:18083:18083
    volumes:
      - monero-wallet-rpc-data:/home/monero
    command:
      - --daemon-address=monerod:18089
      - --trusted-daemon
      - --rpc-bind-port=18083
      - --wallet-dir /home/monero
  moneroblock:
    image: sethsimmons/moneroblock:latest
    restart: unless-stopped
    container_name: moneroblock
    ports:
      - 127.0.0.1:31312:31312
    command:
      - --daemon
      - monerod:18089
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    environment:
      WATCHTOWER_CLEANUP: true
      WATCHTOWER_POLL_INTERVAL: 3600
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  autoheal:
    image: willfarrell/autoheal:latest
    container_name: autoheal
    restart: unless-stopped
    environment:
      AUTOHEAL_CONTAINER_LABEL: all
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
volumes:
  bitmonero: {}
  p2pool-data: {}
  monero-wallet-rpc-data: {}

dearkafka commented 2 days ago

@ubuntophe I feel like you have stated --max-log-files 2 times (my guess)

dearkafka commented 2 days ago

yep, you have

ubuntophe commented 2 days ago

I deleted the last line. but still no luck

hundehausen / monero-suite

Monerod Container starting but never running #12