failed to dial proxy server

[FTwOoO]

server: Alicloud ECS 2 core 4G memory ubuntu 20.04
region: HongKong China docker image: build with Dockerfile of project at branch 1.1.105

run with command:

docker run -d --publish 2154:2154/tcp --name etl --mount type=bind,source=$HOME/etl_data,target=/var/data -e DATABASE_URL=postgresql:xx@xxx.pg.rds.aliyuncs.com:1921/helium_blockchain  registry.cn-shenzhen.aliyuncs.com/risinghf-mirror/helium-blockchain-etl

error:

2021-08-04 08:32:41 =CRASH REPORT====
  crasher:
    initial call: be_db_worker:init/1
    pid: <0.1503.0>
    registered_name: []
    exception error: {{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [<0.1300.0>,db_pool,be_sup,<0.1229.0>]
    message_queue_len: 0
    messages: []
    links: [<0.1300.0>,<0.1504.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 27
    reductions: 303
  neighbours:
2021-08-04 08:32:41 =ERROR REPORT====
** Generic server <0.1300.0> terminating 
** Last message in was {'EXIT',<0.1299.0>,{{badmatch,{error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},[{poolboy,new_worker,1,[{file,"poolboy.erl"},{line,283}]},{poolboy,prepopulate,3,[{file,"poolboy.erl"},{line,304}]},{poolboy,init,3,[{file,"poolboy.erl"},{line,153}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}
** When Server state == {state,{<0.1300.0>,poolboy_sup},simple_one_for_one,{[be_db_worker],#{be_db_worker => {child,undefined,be_db_worker,{be_db_worker,start_link,[[{db_opts,#{database => "helium_blockchain",host => "pgm-wz99m45y36328f605o.pg.rds.aliyuncs.com",password => "S9FJottxCaWJOAPy",port => 1921,username => "dbadmin"}},{db_handlers,[be_db_block,be_db_txn_actor,be_db_account,be_db_gateway,be_db_pending_txn,be_db_geocoder,be_db_gateway_status,be_db_oracle_price,be_db_vars,be_db_stats,be_db_reward,be_db_packet,be_db_validator,be_db_validator_status,be_db_oui,be_db_dc_burn]}]]},temporary,5000,worker,[be_db_worker]}}},{sets,{set,17,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[<0.1475.0>],[<0.1444.0>],[<0.1461.0>,<0.1301.0>],[<0.1430.0>,<0.1414.0>,<0.1382.0>],[<0.1463.0>],[],[<0.1497.0>],[<0.1466.0>],[],[],[<0.1501.0>,<0.1485.0>,<0.1421.0>],[],[<0.1487.0>,<0.1423.0>],[],[<0.1425.0>],[<0.1442.0>]}}}},0,1,[],0,poolboy_sup,{be_db_worker,[{db_opts,#{database => "helium_blockchain",host => "pgm-wz99m45y36328f605o.pg.rds.aliyuncs.com",password => "S9FJottxCaWJOAPy",port => 1921,username => "dbadmin"}},{db_handlers,[be_db_block,be_db_txn_actor,be_db_account,be_db_gateway,be_db_pending_txn,be_db_geocoder,be_db_gateway_status,be_db_oracle_price,be_db_vars,be_db_stats,be_db_reward,be_db_packet,be_db_validator,be_db_validator_status,be_db_oui,be_db_dc_burn]}]}}
** Reason for termination ==
** {{badmatch,{error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},[{poolboy,new_worker,1,[{file,"poolboy.erl"},{line,283}]},{poolboy,prepopulate,3,[{file,"poolboy.erl"},{line,304}]},{poolboy,init,3,[{file,"poolboy.erl"},{line,153}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
2021-08-04 08:32:41 =CRASH REPORT====
  crasher:
    initial call: supervisor:poolboy_sup/1
    pid: <0.1300.0>
    registered_name: []
    exception exit: {{{badmatch,{error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},[{poolboy,new_worker,1,[{file,"poolboy.erl"},{line,283}]},{poolboy,prepopulate,3,[{file,"poolboy.erl"},{line,304}]},{poolboy,init,3,[{file,"poolboy.erl"},{line,153}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},[{gen_server,decode_msg,9,[{file,"gen_server.erl"},{line,432}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [db_pool,be_sup,<0.1229.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.1503.0>,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}]
    links: []
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 2586
    stack_size: 27
    reductions: 3381
  neighbours:
2021-08-04 08:32:41 =CRASH REPORT====
  crasher:
    initial call: application_master:init/4
    pid: <0.1228.0>
    registered_name: []
    exception exit: {{{shutdown,{failed_to_start_child,db_pool,{{badmatch,{error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},[{poolboy,new_worker,1,[{file,"poolboy.erl"},{line,283}]},{poolboy,prepopulate,3,[{file,"poolboy.erl"},{line,304}]},{poolboy,init,3,[{file,"poolboy.erl"},{line,153}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},{blockchain_etl_app,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}
    ancestors: [<0.1227.0>]
    message_queue_len: 1
    messages: [{'EXIT',<0.1229.0>,normal}]
    links: [<0.1227.0>,<0.1036.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 27
    reductions: 243
  neighbours:

==> /root/etl_data/log/error.log <==
2021-08-04 08:13:52.087 [error] <0.1358.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/1125oaysser8zbZGnqPxra4Nnc8LxrLVo7RpLUUvWNCnGJyqhuP5" not_found
2021-08-04 08:14:12.115 [error] <0.1415.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/112scJMeTLK7vUnzVtWMedBGbjVTt1zFRGBQ1ircooMQPS7bXqaX" not_found
2021-08-04 08:17:27.061 [error] <0.1547.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/112ei1wVCANYs8uDQNEY5jSxy3DigswTaEh4vzBiPejgWZALD9Ni" not_found
2021-08-04 08:17:47.846 [error] <0.1326.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/112Qx1coATg7vjkCorNekWBD1P8TAWSUzMfubKGtMV6iXUnEHHvn" not_found
2021-08-04 08:17:47.847 [error] <0.1329.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/11m84qCXH1CdUGN3zpwTN1XW4vchpeYz9ZM5HqvrScwAaNnfyVG" not_found
2021-08-04 08:19:27.206 [error] <0.1443.0> Supervisor {<0.1443.0>,libp2p_simple_sup} had child 1 started with {libp2p_yamux_stream,open_stream,undefined} at <0.1447.0> exit with reason noproc in context shutdown_error
2021-08-04 08:27:17.110 [error] <0.1698.0> gen_server <0.1698.0> terminated with reason: timeout
2021-08-04 08:27:17.111 [error] <0.1230.0>@poolboy:new_worker:283 Supervisor be_sup had child db_pool started with poolboy:start_link([{name,{local,db_pool}},{worker_module,be_db_worker},{size,50},{max_overflow,20}], [{db_opts,#{database => "helium_blockchain",host => "pgm-wz99m45y36328f605o.pg.rds.aliyuncs.co...",...}},...]) at undefined exit with reason no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283 in context start_error
2021-08-04 08:27:17.111 [error] <0.1698.0> CRASH REPORT Process <0.1698.0> with 0 neighbours exited with reason: timeout in gen_server:handle_common_reply/8 line 751
2021-08-04 08:27:17.112 [error] <0.1299.0>@poolboy:new_worker:283 CRASH REPORT Process <0.1299.0> with 0 neighbours crashed with reason: no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283
2021-08-04 08:27:17.112 [error] <0.1697.0>@be_db_worker:init:105 CRASH REPORT Process <0.1697.0> with 0 neighbours crashed with reason: no match of right hand value {error,timeout} in be_db_worker:init/1 line 105
2021-08-04 08:27:17.113 [error] <0.1300.0>@poolboy:new_worker:283 gen_server <0.1300.0> terminated with reason: no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283
2021-08-04 08:27:17.114 [error] <0.1300.0>@poolboy:new_worker:283 CRASH REPORT Process <0.1300.0> with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283 in gen_server:decode_msg/9 line 432
2021-08-04 08:27:17.116 [error] <0.1228.0> CRASH REPORT Process <0.1228.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,db_pool,{{badmatch,{error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},[{poolboy,new_worker,1,[{file,"poolboy.erl"},{line,283}]},{poolboy,prepopulate,3,[{file,"poolboy.erl"},{line,304}]},{poolboy,init,3,[{file,"poolboy.erl"},{line,...}]},...]}}},...} in application_master:init/4 line 138
2021-08-04 08:29:25.651 [error] <0.1322.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/112V7E81QmHxBUQjd4toTSF9d2MV3FiwTNeZQEz2mjCdgjk9K4Z4" not_found
2021-08-04 08:29:25.655 [error] <0.1324.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/1125oaysser8zbZGnqPxra4Nnc8LxrLVo7RpLUUvWNCnGJyqhuP5" not_found
2021-08-04 08:32:41.903 [error] <0.1504.0> gen_server <0.1504.0> terminated with reason: timeout
2021-08-04 08:32:41.904 [error] <0.1230.0>@poolboy:new_worker:283 Supervisor be_sup had child db_pool started with poolboy:start_link([{name,{local,db_pool}},{worker_module,be_db_worker},{size,50},{max_overflow,20}], [{db_opts,#{database => "helium_blockchain",host => "pgm-wz99m45y36328f605o.pg.rds.aliyuncs.co...",...}},...]) at undefined exit with reason no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283 in context start_error
2021-08-04 08:32:41.904 [error] <0.1299.0>@poolboy:new_worker:283 CRASH REPORT Process <0.1299.0> with 0 neighbours crashed with reason: no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283
2021-08-04 08:32:41.904 [error] <0.1504.0> CRASH REPORT Process <0.1504.0> with 0 neighbours exited with reason: timeout in gen_server:handle_common_reply/8 line 751
2021-08-04 08:32:41.904 [error] <0.1503.0>@be_db_worker:init:105 CRASH REPORT Process <0.1503.0> with 0 neighbours crashed with reason: no match of right hand value {error,timeout} in be_db_worker:init/1 line 105
2021-08-04 08:32:41.905 [error] <0.1300.0>@poolboy:new_worker:283 gen_server <0.1300.0> terminated with reason: no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283
2021-08-04 08:32:41.905 [error] <0.1300.0>@poolboy:new_worker:283 CRASH REPORT Process <0.1300.0> with 0 neighbours exited with reason: no match of right hand value {error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}} in poolboy:new_worker/1 line 283 in gen_server:decode_msg/9 line 432
2021-08-04 08:32:41.908 [error] <0.1228.0> CRASH REPORT Process <0.1228.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,db_pool,{{badmatch,{error,{{badmatch,{error,timeout}},[{be_db_worker,init,1,[{file,"be_db_worker.erl"},{line,105}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}},[{poolboy,new_worker,1,[{file,"poolboy.erl"},{line,283}]},{poolboy,prepopulate,3,[{file,"poolboy.erl"},{line,304}]},{poolboy,init,3,[{file,"poolboy.erl"},{line,...}]},...]}}},...} in application_master:init/4 line 138

[madninja]

The key error is here:

{error,{{badmatch,{error,timeout}},[{be_db_worker,init

which implies that the db_worker had issues connecting with your database

[FTwOoO]

The key error is here:

{error,{{badmatch,{error,timeout}},[{be_db_worker,init

which implies that the db_worker had issues connecting with your database

"had issues connecting with your database" is fiexed, now logging: "failed to dial proxy server "/p2p/1128ipxTAurFrUHuXx3XjayKQ75BPXgVWcCUDmTH3CnFJ8HoZy1q" not_found" and also has some success log: "new block sync starting with Pid: <0.1941.0>, Ref: #Ref<0.4078392510.943980545.51030>, Peer: "/p2p/1126uhuuGhDSrrFpxEvV5xURN2xKab6XpTrD5YHcxHVBeMUoN3V4"

is the etl runs well? i run the https://github.com/helium/blockchain-http service and call the stats API:

curl x.x.x.x:80/v1/stats | jq

get null data fields:

{ "data": { "token_supply": 0, "election_times": { "last_week": { "stddev": null, "avg": null },

is the API service runs well ?

[madninja]

The key error is here: {error,{{badmatch,{error,timeout}},[{be_db_worker,init which implies that the db_worker had issues connecting with your database

"had issues connecting with your database" is fiexed, now logging: "failed to dial proxy server "/p2p/1128ipxTAurFrUHuXx3XjayKQ75BPXgVWcCUDmTH3CnFJ8HoZy1q" not_found" and also has some success log: "new block sync starting with Pid: <0.1941.0>, Ref: #Ref<0.4078392510.943980545.51030>, Peer: "/p2p/1126uhuuGhDSrrFpxEvV5xURN2xKab6XpTrD5YHcxHVBeMUoN3V4"

Yes etl is trying to sync with nodes.. It will take a while for it to discover peers to sync blocks with.

is the etl runs well? i run the https://github.com/helium/blockchain-http service and call the stats API:

try :80/v1/blocks/height.

I'm going to guess it will start storing data shortly if you took the blockchain.db dump.

To avoid conversations in multiple places, please join us on discord and ask there since GH issues is not a great place for support questions

[FTwOoO]

The key error is here: {error,{{badmatch,{error,timeout}},[{be_db_worker,init which implies that the db_worker had issues connecting with your database

"had issues connecting with your database" is fiexed, now logging: "failed to dial proxy server "/p2p/1128ipxTAurFrUHuXx3XjayKQ75BPXgVWcCUDmTH3CnFJ8HoZy1q" not_found" and also has some success log: "new block sync starting with Pid: <0.1941.0>, Ref: #Ref<0.4078392510.943980545.51030>, Peer: "/p2p/1126uhuuGhDSrrFpxEvV5xURN2xKab6XpTrD5YHcxHVBeMUoN3V4"

Yes etl is trying to sync with nodes.. It will take a while for it to discover peers to sync blocks with.

is the etl runs well? i run the https://github.com/helium/blockchain-http service and call the stats API:

try :80/v1/blocks/height.

I'm going to guess it will start storing data shortly if you took the blockchain.db dump.

To avoid conversations in multiple places, please join us on discord and ask there since GH issues is not a great place for support questions

thanks, the problem is no problem. there is no way to bootstrap an etl quickly, and the libp2p complains too many error which can be ignore. nobody can tell which is the real problem and which is not, please write some documentation for the project, about architecture/common issue