Closed APshenkin closed 2 years ago
tried to recover from latest snapshot, but issue is still exist
tried to recover from latest snapshot, but issue is still exist
you sure you tried? I'm pretty sure you can't load a snapshot on the cli in blockchain-node so you did something else?
I did cleanup the state of node to get fresh snapshot
I did cleanup the state of node to get fresh snapshot
how? and what snapshot did it install?
how? and what snapshot did it install?
Just moved state to other folder.
Then after docker container reboot
2022-06-30 19:50:23.718 [info] <0.1392.0>@blockchain_worker:do_snap_source_download:1295 Attempting snapshot download from "https://snapshots.helium.wtf/mainnet/snap-1420561.gz", writing to scratch file "/var/data/snap/snap-1420561.gz.scratch"
2022-06-30 19:51:09.053 [info] <0.1329.0>@blockchain:init_blessed_snapshot:2661 Got snapshot for height 1420561 - attempting install
2022-06-30 19:51:09.404 [info] <0.1329.0>@blockchain:init_blessed_snapshot:2669 Snapshot height 1420510, lagging ledger height 1, leading ledger height 1
2022-06-30 19:51:09.406 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 317768
2022-06-30 19:51:09.744 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger:567 loading snapshot into delayed ledger
2022-06-30 19:51:09.745 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 321904
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded consensus_members from snapshot in 310 ms
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 322576
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded election_height from snapshot in 0 ms
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 351088
2022-06-30 19:51:10.377 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded election_epoch from snapshot in 322 ms
2022-06-30 19:51:10.377 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 329688
2022-06-30 19:51:10.698 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:634 loaded delayed_vars from snapshot in 321 ms
2022-06-30 19:51:10.703 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 345096
2022-06-30 19:51:12.085 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:634 loaded threshold_txns from snapshot in 1382 ms
2022-06-30 19:51:12.085 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 369240
2022-06-30 19:51:13.707 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded master_key from snapshot in 1622 ms
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 312488
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded multi_keys from snapshot in 0 ms
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 315336
ready to provide full log if it will help scratch_1624.txt
Also if complete the full fresh launch blockchain-node starts infinitely produce logs like this
2022-06-30 21:19:29.274 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7279.0>, Ref: #Ref<0.292262878.1006895107.202831>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.287 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7280.0>, Ref: #Ref<0.292262878.1006895107.202849>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.302 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7281.0>, Ref: #Ref<0.292262878.1007157249.120946>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.316 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7282.0>, Ref: #Ref<0.292262878.1007157252.43104>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.333 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7283.0>, Ref: #Ref<0.292262878.1006895107.202857>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.342 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7284.0>, Ref: #Ref<0.292262878.1006895107.202863>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.351 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7285.0>, Ref: #Ref<0.292262878.1006895107.202875>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.361 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7286.0>, Ref: #Ref<0.292262878.1006895107.202894>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.371 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7287.0>, Ref: #Ref<0.292262878.1006895107.202896>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.379 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7288.0>, Ref: #Ref<0.292262878.1006895107.202903>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.389 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7289.0>, Ref: #Ref<0.292262878.1006895107.202915>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.405 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7290.0>, Ref: #Ref<0.292262878.1006895106.127977>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.418 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7291.0>, Ref: #Ref<0.292262878.1007157249.120984>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.431 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7292.0>, Ref: #Ref<0.292262878.1007157249.121000>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.443 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7293.0>, Ref: #Ref<0.292262878.1006895106.127999>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.461 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7294.0>, Ref: #Ref<0.292262878.1006895106.128017>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.479 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7295.0>, Ref: #Ref<0.292262878.1007157252.43137>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.488 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7296.0>, Ref: #Ref<0.292262878.1007157252.43148>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.504 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7297.0>, Ref: #Ref<0.292262878.1006895107.202935>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.519 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7298.0>, Ref: #Ref<0.292262878.1006895107.202944>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.535 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7299.0>, Ref: #Ref<0.292262878.1006895107.202963>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.550 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7300.0>, Ref: #Ref<0.292262878.1007157249.121011>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.560 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7301.0>, Ref: #Ref<0.292262878.1007157249.121022>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
In previous set up we tried to swap peerbook
from old version that was backed up
Finally after switching to docker images provided by you from https://quay.io/repository/team-helium/blockchain-node?tab=tags
BTW not clear for me why our images builded from source didn't work
Here is our Dockerfile
FROM erlang:22.3.2-alpine as builder
ARG CONFIG
RUN apk add --no-cache --update \
git tar build-base linux-headers autoconf automake libtool pkgconfig \
dbus-dev bzip2 bison flex gmp-dev cmake lz4 libsodium-dev openssl-dev \
sed curl cargo
# Install Rust toolchain
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
WORKDIR /usr/src/node
ENV CC=gcc CXX=g++ CFLAGS="-U__sun__" \
ERLANG_ROCKSDB_OPTS="-DWITH_BUNDLE_SNAPPY=ON -DWITH_BUNDLE_LZ4=ON" \
ERL_COMPILER_OPTIONS="[deterministic]" \
PATH="/root/.cargo/bin:$PATH" \
RUSTFLAGS="-C target-feature=-crt-static"
ARG TAG=1.1.66
# Add our code
RUN git clone https://github.com/helium/blockchain-node.git /usr/src/node && git checkout $TAG
# copy testnet config
COPY . /usr/src/node
RUN ./rebar3 as $CONFIG tar
RUN mkdir -p /opt/docker
RUN tar -zxvf _build/$CONFIG/rel/*/*.tar.gz -C /opt/docker
RUN mkdir -p /opt/docker/update
FROM erlang:22.3.2-alpine as runner
RUN apk add --no-cache --update ncurses dbus gmp libsodium gcc
RUN ulimit -n 64000
WORKDIR /opt/node
ENV COOKIE=node \
# Write files generated during startup to /tmp
RELX_OUT_FILE_PATH=/tmp \
# add miner to path, for easy interactions
PATH=$PATH:/opt/node/bin
COPY --from=builder /opt/docker /opt/node
ENTRYPOINT ["/opt/node/bin/blockchain_node"]
CMD ["foreground"]
Any thoughts?
We've found the bug is related to OTP22, use an OTP24 builder in your dockerfile
This issue belongs in the blockchain-node repo since that’s where the Dockerfile lives. It was addressed there already
We updated
blockchain-node
to 1.1.66 two days ago before todays hard forkBut starting 18:00 UTC time syncronization stops. Get a lot of errors in logs for all nodes