APshenkin
commented
2 years ago tried to recover from latest snapshot, but issue is still exist
Closed APshenkin closed 2 years ago
APshenkin
commented
2 years ago tried to recover from latest snapshot, but issue is still exist
madninja
commented
2 years ago tried to recover from latest snapshot, but issue is still exist
you sure you tried? I'm pretty sure you can't load a snapshot on the cli in blockchain-node so you did something else?
APshenkin
commented
2 years ago I did cleanup the state of node to get fresh snapshot
madninja
commented
2 years ago I did cleanup the state of node to get fresh snapshot
how? and what snapshot did it install?
APshenkin
commented
2 years ago how? and what snapshot did it install?
Just moved state to other folder.
Then after docker container reboot
2022-06-30 19:50:23.718 [info] <0.1392.0>@blockchain_worker:do_snap_source_download:1295 Attempting snapshot download from "https://snapshots.helium.wtf/mainnet/snap-1420561.gz", writing to scratch file "/var/data/snap/snap-1420561.gz.scratch"
2022-06-30 19:51:09.053 [info] <0.1329.0>@blockchain:init_blessed_snapshot:2661 Got snapshot for height 1420561 - attempting install
2022-06-30 19:51:09.404 [info] <0.1329.0>@blockchain:init_blessed_snapshot:2669 Snapshot height 1420510, lagging ledger height 1, leading ledger height 1
2022-06-30 19:51:09.406 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 317768
2022-06-30 19:51:09.744 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger:567 loading snapshot into delayed ledger
2022-06-30 19:51:09.745 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 321904
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded consensus_members from snapshot in 310 ms
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 322576
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded election_height from snapshot in 0 ms
2022-06-30 19:51:10.055 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 351088
2022-06-30 19:51:10.377 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded election_epoch from snapshot in 322 ms
2022-06-30 19:51:10.377 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 329688
2022-06-30 19:51:10.698 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:634 loaded delayed_vars from snapshot in 321 ms
2022-06-30 19:51:10.703 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 345096
2022-06-30 19:51:12.085 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:634 loaded threshold_txns from snapshot in 1382 ms
2022-06-30 19:51:12.085 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 369240
2022-06-30 19:51:13.707 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded master_key from snapshot in 1622 ms
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 312488
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:load_into_ledger_:648 loaded multi_keys from snapshot in 0 ms
2022-06-30 19:51:13.708 [info] <0.1329.0>@blockchain_ledger_snapshot_v1:print_memory:653 memory {total_heap_size,2586} 315336ready to provide full log if it will help scratch_1624.txt
APshenkin
commented
2 years ago Also if complete the full fresh launch blockchain-node starts infinitely produce logs like this
2022-06-30 21:19:29.274 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7279.0>, Ref: #Ref<0.292262878.1006895107.202831>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.287 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7280.0>, Ref: #Ref<0.292262878.1006895107.202849>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.302 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7281.0>, Ref: #Ref<0.292262878.1007157249.120946>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.316 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7282.0>, Ref: #Ref<0.292262878.1007157252.43104>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.333 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7283.0>, Ref: #Ref<0.292262878.1006895107.202857>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.342 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7284.0>, Ref: #Ref<0.292262878.1006895107.202863>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.351 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7285.0>, Ref: #Ref<0.292262878.1006895107.202875>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.361 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7286.0>, Ref: #Ref<0.292262878.1006895107.202894>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.371 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7287.0>, Ref: #Ref<0.292262878.1006895107.202896>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.379 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7288.0>, Ref: #Ref<0.292262878.1006895107.202903>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.389 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7289.0>, Ref: #Ref<0.292262878.1006895107.202915>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.405 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7290.0>, Ref: #Ref<0.292262878.1006895106.127977>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.418 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7291.0>, Ref: #Ref<0.292262878.1007157249.120984>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.431 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7292.0>, Ref: #Ref<0.292262878.1007157249.121000>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.443 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7293.0>, Ref: #Ref<0.292262878.1006895106.127999>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.461 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7294.0>, Ref: #Ref<0.292262878.1006895106.128017>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.479 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7295.0>, Ref: #Ref<0.292262878.1007157252.43137>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.488 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7296.0>, Ref: #Ref<0.292262878.1007157252.43148>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.504 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7297.0>, Ref: #Ref<0.292262878.1006895107.202935>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.519 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7298.0>, Ref: #Ref<0.292262878.1006895107.202944>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.535 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7299.0>, Ref: #Ref<0.292262878.1006895107.202963>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.550 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7300.0>, Ref: #Ref<0.292262878.1007157249.121011>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"
2022-06-30 21:19:29.560 [info] <0.1354.0>@blockchain_worker:start_sync:{897,13} new block sync starting with Pid: <0.7301.0>, Ref: #Ref<0.292262878.1007157249.121022>, Peer: "/p2p/112RDB1K4aP8CSmwPmozFYhrjfqsmimctx9Svz4PZ3cFzhbDX3Sy"In previous set up we tried to swap peerbook from old version that was backed up
APshenkin
commented
2 years ago Finally after switching to docker images provided by you from https://quay.io/repository/team-helium/blockchain-node?tab=tags
BTW not clear for me why our images builded from source didn't work
Here is our Dockerfile
FROM erlang:22.3.2-alpine as builder
ARG CONFIG
RUN apk add --no-cache --update \
git tar build-base linux-headers autoconf automake libtool pkgconfig \
dbus-dev bzip2 bison flex gmp-dev cmake lz4 libsodium-dev openssl-dev \
sed curl cargo
# Install Rust toolchain
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
WORKDIR /usr/src/node
ENV CC=gcc CXX=g++ CFLAGS="-U__sun__" \
ERLANG_ROCKSDB_OPTS="-DWITH_BUNDLE_SNAPPY=ON -DWITH_BUNDLE_LZ4=ON" \
ERL_COMPILER_OPTIONS="[deterministic]" \
PATH="/root/.cargo/bin:$PATH" \
RUSTFLAGS="-C target-feature=-crt-static"
ARG TAG=1.1.66
# Add our code
RUN git clone https://github.com/helium/blockchain-node.git /usr/src/node && git checkout $TAG
# copy testnet config
COPY . /usr/src/node
RUN ./rebar3 as $CONFIG tar
RUN mkdir -p /opt/docker
RUN tar -zxvf _build/$CONFIG/rel/*/*.tar.gz -C /opt/docker
RUN mkdir -p /opt/docker/update
FROM erlang:22.3.2-alpine as runner
RUN apk add --no-cache --update ncurses dbus gmp libsodium gcc
RUN ulimit -n 64000
WORKDIR /opt/node
ENV COOKIE=node \
# Write files generated during startup to /tmp
RELX_OUT_FILE_PATH=/tmp \
# add miner to path, for easy interactions
PATH=$PATH:/opt/node/bin
COPY --from=builder /opt/docker /opt/node
ENTRYPOINT ["/opt/node/bin/blockchain_node"]
CMD ["foreground"]Any thoughts?
Vagabond
commented
2 years ago We've found the bug is related to OTP22, use an OTP24 builder in your dockerfile
madninja
commented
2 years ago This issue belongs in the blockchain-node repo since that’s where the Dockerfile lives. It was addressed there already
We updated
blockchain-nodeto 1.1.66 two days ago before todays hard forkBut starting 18:00 UTC time syncronization stops. Get a lot of errors in logs for all nodes