search

helium / miner

Miner for the helium blockchain
Apache License 2.0
607 stars 265 forks source link

Please help me understand what breaks with this #1901

Open teodoryantcheff opened 1 year ago

teodoryantcheff commented 1 year ago

So, I have this Pisces P100 miner that has been working completely ok for the better part of last year. A month ago miner's dashboard started showing blank witness logs, miner height is also blank, it's constantly syncing :

image

image

yet it seems to be mining ok.

Tried clearing blockchain data, restarting, resetting - still the same.

This is what I can see, logging on the miner:

CONTAINER ID        IMAGE                                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
752ab769951f        quay.io/team-helium/miner:miner-arm64_2023.02.07.0_GA   "/opt/miner/bin/mine…"   4 weeks ago         Up 2 weeks                              miner
099a33cf462d        gateway_mfr:arm64                                       "/opt/gateway_mfr/bi…"   19 months ago       Up 3 weeks                              provision
raspberrypi:~#
raspberrypi:~# docker logs miner
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
dets: file "/var/data/blockchain_swarm/cache.dets" not properly closed, repairing ...
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Exec: /opt/miner/erts-12.3.2.7/bin/erlexec -noinput +Bd -boot /opt/miner/releases/2023.02.07.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/miner/lib -config /opt/miner/releases/2023.02.07.0/sys.config -args_file /opt/miner/releases/2023.02.07.0/vm.args -- foreground
Root: /opt/miner
/opt/miner
thread '<unnamed>' panicked at 'slog::Fuse Drain: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/slog-2.7.0/src/lib.rs:1944:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
raspberrypi:~#

But I have no idea how to make sense of this log.

Please help me understand what's breaking and advise on how to proceed.

Many thanks in advance.

teodoryantcheff commented 1 year ago

Digging a bit more into this lead me to: 

admin@raspberrypi:~ $ sudo docker exec miner miner info height
RPC to 'miner@127.0.0.1' failed: {'EXIT',
                                  {function_clause,
                                   [{blockchain,sync_height,
                                     [undefined],
                                     [{file,"blockchain.erl"},{line,521}]},
                                    {miner_cli_info,get_info,0,
                                     [{file,"miner_cli_info.erl"},{line,100}]},
                                    {miner_cli_info,info_height,3,
                                     [{file,"miner_cli_info.erl"},{line,105}]},
                                    {clique_command,run,1,
                                     [{file,"clique_command.erl"},{line,87}]},
                                    {blockchain_console,command,1,
                                     [{file,"blockchain_console.erl"},
                                      {line,14}]}]}}
admin@raspberrypi:~ $

I also saw lots of warnings in the logs: 

admin@raspberrypi:~ $ sudo docker exec miner tail -F /var/data/log/console.log | grep warning
2023-03-20 16:14:36.446 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:36.746 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:38.130 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:38.431 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:40.902 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:41.202 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:42.120 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:42.421 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:48.754 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:49.054 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:14:51.764 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:14:52.065 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:01.573 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:01.875 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:12.919 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:13.219 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:13.225 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:13.565 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:15.157 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:15.458 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:17.709 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:18.010 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:23.886 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:24.187 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router
2023-03-20 16:15:24.371 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 16, uri: http://13.37.13.24:8080/, pubkey: 11afuQSrmk52mgxLu91AdtDXbJ9wmqWBUxC3hvjejoXkxEZfPvY, module: router
2023-03-20 16:15:24.672 6 [warning] <0.1643.0>@miner_gateway_port:dispatch_port_logs:{197,13} [ gateway-rs ] ignoring failed uplink Service(Rpc(Status { code: Unknown, message: "routing_not_found", metadata: MetadataMap { headers: {"user-agent": "grpc-erlang/0.1.0", "content-type": "application/grpc+proto", "grpc-encoding": "identity"} }, source: None })), oui: 9, uri: http://44.238.156.97:8080/, pubkey: 11w77YQLhgUt8HUJrMtntGGr97RyXmot1ofs5Ct2ELTmbFoYsQa, module: router

And the peer list is ...empty:

admin@raspberrypi:~ $ sudo docker exec miner miner peer book -s
+---------------------------------------------+--------------+----------+---------+------+----------+
|                   address                   |     name     |listen_add|connectio| nat  |last_updat|
+---------------------------------------------+--------------+----------+---------+------+----------+
|/p2p/112ECz8yR9Q3QzJJZQyafX7gZzxmx6nzGTYggLVk|able-sangria-c|    0     |    0    |unknow| 64.022s  |
+---------------------------------------------+--------------+----------+---------+------+----------+

admin@raspberrypi:~ $

Trying to list snapshots:

RPC to 'miner@127.0.0.1' failed: {'EXIT',
                                  {function_clause,
                                   [{blockchain,head_hash,
                                     [undefined],
                                     [{file,"blockchain.erl"},{line,448}]},
                                    {blockchain,head_block,1,
                                     [{file,"blockchain.erl"},{line,487}]},
                                    {blockchain,find_last_snapshots,2,
                                     [{file,"blockchain.erl"},{line,2241}]},
                                    {blockchain_cli_snapshot,snapshot_list,3,
                                     [{file,"blockchain_cli_snapshot.erl"},
                                      {line,214}]},
                                    {clique_command,run,1,
                                     [{file,"clique_command.erl"},{line,87}]},
                                    {blockchain_console,command,1,
                                     [{file,"blockchain_console.erl"},
                                      {line,14}]}]}}
admin@raspberrypi:~ $

Following the maker guide from here, LoRa checks show:

admin@raspberrypi:~ $ sudo docker exec miner tail -f /var/data/log/console.log | grep lora
2023-03-20 16:21:39.791 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 2,<<"codr">> => <<"4/5">>,<<"data">> => <<"gKqq8hCAbgAB2UpLDxOWhV29j9jsLrMj6qNpoAqx9eszgq18MgIc3U9cIzCyGkJDtS425RDq12uaaxk98zXKJQ==">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.5,<<"lsnr">> => -3.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -106,<<"rssis">> => -110,<<"size">> => 64,<<"stat">> => 1,<<"tmst">> => 3185641417}]} from 12273815315514654720 on 53338
2023-03-20 16:21:40.631 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{350,5} PULL_DATA from 12273815315514654720 on 53338
2023-03-20 16:21:46.897 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 1,<<"codr">> => <<"4/5">>,<<"data">> => <<"QIfnBAKAVAAO9UFWj1hWi2lYv36Buajl/60bs+Lt">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.3,<<"lsnr">> => -8.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -106,<<"rssis">> => -113,<<"size">> => 30,<<"stat">> => 1,<<"tmst">> => 3192754773}]} from 12273815315514654720 on 53338
2023-03-20 16:21:47.132 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 6,<<"codr">> => <<"4/5">>,<<"data">> => <<"4Ht07Tt7BUx/Y5HjuCtVFC8EDSkYm0vKs+gm1WOMV+EcxbnqmxHeEpsWz1QiXccl3NV3kA==">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 867.7,<<"lsnr">> => -1.2,<<"modu">> => <<"LORA">>,<<"rfch">> => 0,<<"rssi">> => -97,<<"rssis">> => -100,<<"size">> => 52,<<"stat">> => 1,<<"tmst">> => 3192988611}]} from 12273815315514654720 on 53338
2023-03-20 16:21:50.632 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{350,5} PULL_DATA from 12273815315514654720 on 53338
2023-03-20 16:22:00.633 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{350,5} PULL_DATA from 12273815315514654720 on 53338
2023-03-20 16:22:00.886 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 0,<<"codr">> => <<"4/5">>,<<"data">> => <<"gB9zEBCAwgABQAEXnBkIMHuG1/qiYUJnVU7zpfiwxuMkcheZcx1J3aYfGM50LUk/rLmAWGUy8b93PM+o9kXJEg==">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.1,<<"lsnr">> => -8.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -110,<<"rssis">> => -117,<<"size">> => 64,<<"stat">> => 1,<<"tmst">> => 3206742901}]} from 12273815315514654720 on 53338
2023-03-20 16:22:05.109 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"rxpk">> => [#{<<"chan">> => 0,<<"codr">> => <<"4/5">>,<<"data">> => <<"AMEorQEAe0kcRFUJFQghWgA+gA0UsJ0=">>,<<"datr">> => <<"SF12BW125">>,<<"freq">> => 868.1,<<"lsnr">> => -9.0,<<"modu">> => <<"LORA">>,<<"rfch">> => 1,<<"rssi">> => -110,<<"rssis">> => -118,<<"size">> => 23,<<"stat">> => 1,<<"tmst">> => 3210966694}]} from 12273815315514654720 on 53338
2023-03-20 16:22:05.832 6 [info] <0.1760.0>@miner_lora_light:handle_udp_packet:{326,5} PUSH_DATA #{<<"stat">> => #{<<"ackr">> => 100.0,<<"dwnb">> => 0,<<"rxfw">> => 6,<<"rxnb">> => 19,<<"rxok">> => 6,<<"time">> => <<"2023-03-20 16:22:05 GMT">>,<<"txnb">> => 0}} from 12273815315514654720 on 53338
teodoryantcheff commented 1 year ago

The last thing that came to mind is to look at the "gateway" container, which seems to be constantly crashed : 

raspberrypi:~# docker logs --since 30m -f provision
=INFO REPORT==== 20-Mar-2023::16:29:25.067585 ===
SIGTERM received - shutting down

=SUPERVISOR REPORT==== 20-Mar-2023::16:29:25.085367 ===
    supervisor: {local,gr_counter_sup}
    errorContext: child_terminated
    reason: killed
    offender: [{pid,<0.464.0>},
               {id,gr_lager_default_tracer_counters},
               {mfargs,{gr_counter,start_link,
                                   [gr_lager_default_tracer_counters]}},
               {restart_type,transient},
               {shutdown,brutal_kill},
               {child_type,worker}]
=SUPERVISOR REPORT==== 20-Mar-2023::16:29:25.086479 ===
    supervisor: {local,gr_param_sup}
    errorContext: child_terminated
    reason: killed
    offender: [{pid,<0.463.0>},
               {id,gr_lager_default_tracer_params},
               {mfargs,{gr_param,start_link,[gr_lager_default_tracer_params]}},
               {restart_type,transient},
               {shutdown,brutal_kill},
               {child_type,worker}]
Exec: /usr/local/lib/erlang/erts-10.7.1/bin/erlexec -noshell -noinput +Bd -boot /opt/gateway_mfr/releases/0.1.0/start -mode embedded -boot_var ERTS_LIB_DIR /usr/local/lib/erlang/lib -config /opt/gateway_mfr/releases/0.1.0/sys.config -args_file /opt/gateway_mfr/releases/0.1.0/vm.args -- foreground
Root: /opt/gateway_mfr
/opt/gateway_mfr