blockchain-etl database not catching up with the helium network

[rameshsraj]

Hi, I am unable to start the helium blockchain process, I am getting the below error. It was working fine , I had to start it as it stopped working(stopped pulling data from the helium network)

Already up to date. ./_build/dev/rel/blockchain_etl/bin/blockchain_etl stop make: [Makefile:52: stop] Error 1 (ignored) ./rebar3 compile ./rebar3 as dev do release ===> Verifying dependencies... ===> Verifying dependencies... ===> Plugin rebar3_eqc does not export init/1. It will not be used. ===> Plugin rebar3_eqc does not export init/1. It will not be used. ===> Analyzing applications... ===> Analyzing applications... ===> Compiling blockchain_etl ===> Compiling blockchain_etl ===> Assembling release blockchain_etl-0.1.0... ===> rendering builtin_hook_wait_for_process hook to _build/dev/rel/blockchain_etl/bin/hooks/builtin/wait_for_process ===> Warnings generating release: WARNING Missing application sasl. Can not upgrade with this release ===> Release successfully assembled: _build/dev/rel/blockchain_etl ./_build/dev/rel/blockchain_etl/bin/blockchain_etl stop make: [Makefile:52: stop] Error 1 (ignored) ./_build/dev/rel/blockchain_etl/bin/blockchain_etl migrations run No migrations were run cp -f '.env' ./_build/dev/rel/blockchain_etl/.env ./_build/dev/rel/blockchain_etl/bin/blockchain_etl daemon

===== ===== LOGGING STARTED Thu Apr 29 18:55:14 UTC 2021 Exec: /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl/erts-10.7.2.1/bin/erlexec -boot /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl/releases/0.1.0/blockchain_etl -mode embedded -boot_var SYSTEM_LIB_DIR /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl/lib -config /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl/releases/0.1.0/sys.config -args_file /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl/releases/0.1.0/vm.args -- console --relx-disable-hooks Root: /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl /home/simh1989/blockchain-etl/_build/dev/rel/blockchain_etl Erlang/OTP 22 [erts-10.7.2.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]

2021-04-29 18:55:15.471569 supervisor_report #{label=>{supervisor,start_error},report=>[{supervisor,{local,kernel_sup}},{errorContext,start_error},{reason,{on_load_function_failed,rocksdb}},{offender,[{pid,undefined},{id,kernel_safe_sup},{mfargs,{supervisor,start_link,[{local,kernel_safe_sup},kernel,safe]}},{restart_type,permanent},{shutdown,infinity},{child_type,supervisor}]}]} 2021-04-29 18:55:15.471620 crash_report #{label=>{proc_lib,crash},report=>[[{initial_call,{supervisor,kernel,['Argument__1']}},{pid,<0.1039.0>},{registered_name,[]},{error_info,{exit,{on_load_function_failed,rocksdb},[{init,run_on_load_handlers,0,[]},{kernel,init,1,[{file,"kernel.erl"},{line,189}]},{supervisor,init,1,[{file,"supervisor.erl"},{line,295}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[kernel_sup,<0.1011.0>]},{message_queue_len,0},{messages,[]},{links,[<0.1013.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,265}],[]]} 2021-04-29 18:55:16.475272 crash_report #{label=>{proc_lib,crash},report=>[[{initial_call,{application_master,init,['Argument1','Argument2','Argument3','Argument4']}},{pid,<0.1010.0>},{registered_name,[]},{error_info,{exit,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,rocksdb}}},{kernel,start,[normal,[]]}},[{application_master,init,4,[{file,"application_master.erl"},{line,138}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}},{ancestors,[<0.1009.0>]},{message_queue_len,1},{messages,[{'EXIT',<0.1011.0>,normal}]},{links,[<0.1009.0>,<0.1008.0>]},{dictionary,[]},{trap_exit,true},{status,running},{heap_size,610},{stack_size,27},{reductions,193}],[]]} 2021-04-29 18:55:16.477500 std_info #{label=>{application_controller,exit},report=>[{application,kernel},{exited,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,rocksdb}}},{kernel,start,[normal,[]]}}},{type,permanent}]} {"Kernel pid terminated",application_controller,"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,rocksdb}}},{kernel,start,[normal,[]]}}}"} Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,kernel_safe_sup,{on_load_function_failed,rocksdb}}},{kernel,start,[normal,[]]}}})

Crash dump is being written to: erl_crash.dump...done

erl_crash.dump. active_tasks active ac absoluteURI abs_path abort 'EXIT' 'UP' 'DOWN' '3RT$' undefined_lambda undefined_function nocatch undefined exit error throw return call normal timeout infinity '' '$end_oftable' 'nonode@nohost' '' true false =end

Thank you.

[rameshsraj]

Reconfigured it today, getting the below message, the local etl database is not catching up

2021-04-30 05:57:35.321 [info] <0.1946.0>@libp2p_transport_proxy:connect:38 init proxy with ["/p2p/11aHq1wrEH7mXv5jZsUaxKbp3fQ9LeMzxGd3YHeGv3Tk6fiXeEj","/p2p/11peaqNCEgTwHyxkSSzjpXiVQBCLBUvetYE5gJWUow2ufnG6K9d"] 2021-04-30 05:57:35.992 [info] <0.1967.0>@libp2p_stream_proxy:init:63 init proxy client with {{connection,libp2p_yamux_stream,<0.1961.0>},[{swarm,<0.1206.0>},{p2p_circuit,"/p2p/11aHq1wrEH7mXv5jZsUaxKbp3fQ9LeMzxGd3YHeGv3Tk6fiXeEj/p2p-circuit/p2p/11peaqNCEgTwHyxkSSzjpXiVQBCLBUvetYE5gJWUow2ufnG6K9d"},{transport,<0.1946.0>},{id,<<216,97,33,102,133,120,213,83,205,216,72,244,145,235,108,61>>}]} 2021-04-30 05:57:36.111 [notice] <0.1213.0>@libp2p_transport_tcp:record_observed_addr:895 recording observed address "/p2p/11aHq1wrEH7mXv5jZsUaxKbp3fQ9LeMzxGd3YHeGv3Tk6fiXeEj" "/ip4/34.121.0.19/tcp/44158" 2021-04-30 05:57:36.277 [info] <0.1967.0>@libp2p_stream_proxy:handle_client_data:174 client got dial back request {libp2p_proxy_dial_back_pb} 2021-04-30 05:57:36.278 [info] <0.1967.0>@libp2p_stream_proxy:dial_back:226 Got port "44158" from peerbook for "/p2p/11aHq1wrEH7mXv5jZsUaxKbp3fQ9LeMzxGd3YHeGv3Tk6fiXeEj" 2021-04-30 05:57:36.542 [info] <0.1967.0>@libp2p_stream_proxy:handle_client_data:188 client got proxy resp {libp2p_proxy_resp_pb,true,<<"/ip4/50.194.62.57/tcp/45087">>} 2021-04-30 05:57:36.543 [info] <0.1946.0>@libp2p_transport_proxy:connect_rcv:110 proxy successful {connection,libp2p_transport_tcp,{tcp_state,{"/ip4/10.128.0.6/tcp/56613","/ip4/50.194.62.57/tcp/45087"},#Port<0.161>,undefined,ranch_tcp}} 2021-04-30 05:57:38.949 [info] <0.1942.0>@blockchain_sync_handler:handle_data:113 adding sync blocks [1157,1158,1159,1160,1161]

[rameshsraj]

I am seeing the below in the logs, but don't see any rows getting updated on the DB side. ...

2021-04-30 07:36:23.269 [info] <0.1334.0>@be_db_pending_txn:handle_info:105 Submitting 0 pending transactions 2021-04-30 07:36:27.906 [info] <0.4531.0>@blockchain_sync_handler:handle_data:113 adding sync blocks [9062,9063,9064,9065,9066] 2021-04-30 07:36:27.906 [info] <0.4552.0>@blockchain:save_temp_block:2006 temp block height is at least 9062 2021-04-30 07:36:27.910 [info] <0.4552.0>@blockchain:save_temp_block:2006 temp block height is at least 9063 2021-04-30 07:36:27.912 [info] <0.4552.0>@blockchain:save_temp_block:2006 temp block height is at least 9064 2021-04-30 07:36:27.915 [info] <0.4552.0>@blockchain:save_temp_block:2006 temp block height is at least 9065 2021-04-30 07:36:27.917 [info] <0.4552.0>@blockchain:save_temp_block:2006 temp block height is at least 9066

1-04-30 07:40:42.205 [info] <0.6802.0>@blockchain:save_temp_block:2006 temp block height is at least 14363 2021-04-30 07:40:42.207 [info] <0.6802.0>@blockchain:save_temp_block:2006 temp block height is at least 14364 2021-04-30 07:40:42.209 [info] <0.6802.0>@blockchain:save_temp_block:2006 temp block height is at least 14365 2021-04-30 07:40:42.211 [info] <0.6802.0>@blockchain:save_temp_block:2006 temp block height is at least 14366 2021-04-30 07:40:42.267 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.6803.0>, Ref: #Ref<0.3409173699.2173173761.41794> 2021-04-30 07:40:42.482 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.6807.0>, Ref: #Ref<0.3409173699.2173173761.41853> 2021-04-30 07:40:42.667 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.6811.0>, Ref: #Ref<0.3409173699.2173173761.41912> 2021-04-30 07:40:42.850 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.6815.0>, Ref: #Ref<0.3409173699.2173173761.41973> 2021-04-30 07:40:42.893 [info] <0.6819.0>@libp2p_transport_proxy:connect:38 init proxy with ["/p2p/11aL3BryVgZDLUyBQezixvr6TSKXPbukArtUX8sZ2yudpKKiNuD","/p2p/113NCyjEnnc3i41SyDMLy6AKkYXmqu9JmcCo4b9hPVLWxpSpRbv"] 2021-04-30 07:40:42.894 [error] <0.6819.0>@libp2p_transport_proxy:connect_to:69 failed to dial proxy server "/p2p/11aL3BryVgZDLUyBQezixvr6TSKXPbukArtUX8sZ2yudpKKiNuD" not_found

2021-04-30 08:54:51.427 [info] <0.1804.0>@blockchain_sync_handler:handle_data:113 adding sync blocks [24172,24173,24174,24175,24176] 2021-04-30 08:54:51.427 [info] <0.1875.0>@blockchain:save_temp_block:2006 temp block height is at least 24172 2021-04-30 08:54:51.431 [info] <0.1875.0>@blockchain:save_temp_block:2006 temp block height is at least 24173 2021-04-30 08:54:51.434 [info] <0.1875.0>@blockchain:save_temp_block:2006 temp block height is at least 24174 2021-04-30 08:54:51.436 [info] <0.1875.0>@blockchain:save_temp_block:2006 temp block height is at least 24175 2021-04-30 08:54:51.439 [info] <0.1875.0>@blockchain:save_temp_block:2006 temp block height is at least 24176 2021-04-30 08:54:51.503 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.1876.0>, Ref: #Ref<0.3430906446.3259760642.126295> 2021-04-30 08:54:54.281 [info] <0.1342.0>@be_db_pending_txn:handle_info:105 Submitting 0 pending transactions 2021-04-30 08:54:55.430 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.1880.0>, Ref: #Ref<0.3430906446.3260547073.80646> 2021-04-30 08:55:04.283 [info] <0.1342.0>@be_db_pending_txn:handle_info:105 Submitting 0 pending transactions 2021-04-30 08:55:08.618 [warning] <0.1225.0>@libp2p_group_worker:connecting:237 Failed to connect to "/ip4/34.208.255.251/tcp/2154": timeout 2021-04-30 08:55:14.284 [info] <0.1342.0>@be_db_pending_txn:handle_info:105 Submitting 0 pending transactions 2021-04-30 08:55:15.788 [notice] <0.1213.0>@libp2p_transport_tcp:record_observed_addr:895 recording observed address "/p2p/112bLxdLCw4VwB3AyhGd5SCqpt8xFQZsFV9oSJ3YwGzxkjEewnP5" "/ip4/34.121.0.19/tcp/44158" 2021-04-30 08:55:18.482 [info] <0.1882.0>@blockchain_sync_handler:handle_data:113 adding sync blocks [24177,24178,24179,24180,24181] 2021-04-30 08:55:18.483 [info] <0.1922.0>@blockchain:save_temp_block:2006 temp block height is at least 24177

[rameshsraj]

**I followed the instruction mentionedhere https://github.com/helium/blockchain-etl

It starts fine, but it inserted some 70 records into the transactions table and then it stopped** -- not much activity, all the sessions are in PARSE state.,

[rameshsraj]

I am seeing the below in the logs, but no activity on the DB side.

2021-04-30 10:24:17.173 [info] <0.32078.0>@libp2p_transport_proxy:connect:38 init proxy with ["/p2p/11RbrFtc7ixSDMfkDJMQNUBmXVaKHKuS4Jqnh2nMs9NtoqfxfiX","/p2p/112vSKFQsqLn6s9iDbacnVVZW6UTiQGExYWeysmwo2CPQwQ8u91f"] 2021-04-30 10:24:17.422 [info] <0.32091.0>@libp2p_stream_proxy:init:63 init proxy client with {{connection,libp2p_yamux_stream,<0.32084.0>},[{swarm,<0.1206.0>},{p2p_circuit,"/p2p/11RbrFtc7ixSDMfkDJMQNUBmXVaKHKuS4Jqnh2nMs9NtoqfxfiX/p2p-circuit/p2p/112vSKFQsqLn6s9iDbacnVVZW6UTiQGExYWeysmwo2CPQwQ8u91f"},{transport,<0.32078.0>},{id,<<143,143,223,66,89,205,28,109,248,74,124,195,38,72,205,178>>}]} 2021-04-30 10:24:17.497 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.32095.0>, Ref: #Ref<0.3430906446.3260022786.4160> 2021-04-30 10:24:17.579 [notice] <0.1213.0>@libp2p_transport_tcp:record_observed_addr:895 recording observed address "/p2p/11RbrFtc7ixSDMfkDJMQNUBmXVaKHKuS4Jqnh2nMs9NtoqfxfiX" "/ip4/34.121.0.19/tcp/44158" 2021-04-30 10:24:17.634 [info] <0.32091.0>@libp2p_stream_proxy:handle_client_data:174 client got dial back request {libp2p_proxy_dial_back_pb} 2021-04-30 10:24:17.635 [info] <0.32091.0>@libp2p_stream_proxy:dial_back:226 Got port "44158" from peerbook for "/p2p/11RbrFtc7ixSDMfkDJMQNUBmXVaKHKuS4Jqnh2nMs9NtoqfxfiX" 2021-04-30 10:24:17.709 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.32099.0>, Ref: #Ref<0.3430906446.3260809217.223833> 2021-04-30 10:24:17.846 [info] <0.32091.0>@libp2p_stream_proxy:handle_client_data:188 client got proxy resp {libp2p_proxy_resp_pb,true,<<"/ip4/69.217.141.61/tcp/58399">>} 2021-04-30 10:24:17.846 [info] <0.32078.0>@libp2p_transport_proxy:connect_rcv:110 proxy successful {connection,libp2p_transport_tcp,{tcp_state,{"/ip4/10.128.0.6/tcp/56477","/ip4/69.217.141.61/tcp/58399"},#Port<0.1156>,undefined,ranch_tcp}} 2021-04-30 10:24:18.278 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.32112.0>, Ref: #Ref<0.3430906446.3260809217.223967> 2021-04-30 10:24:18.875 [info] <0.1242.0>@blockchain_worker:start_sync:739 new sync starting with Pid: <0.32121.0>, Ref: #Ref<0.3430906446.3260809217.224113>

[jamiew]

How long have you been running? It can take multiple days to download the full ledger, and then days to insert it. ETL first syncs ledger then does an import pass. Those logs look like innocuous mostly-libp2p startup messages

[rameshsraj]

I am running for the past 5 to 6 hours

how long it will take it to update the ledger, do we have any timeline .. 2 days 3 days, etc

Your recommendation is to wait?.

[madninja]

It takes weeks to sync from scratch. All the above is normal behavior. You can either wait and check on the box occasionally to make sure it didn't fall over, or join us in discord for a speedup hack that will make it go a bit faster (still takes at least a week even with the speedup).

[rameshsraj]

Thank you @madninja @jamiew , I will join the group, Also, quick check, I have the db files from the old path, will that help in speeding up, if I copy that *.db files into this new location.

[madninja]

Also, quick check, I have the db files from the old path, will that help in speeding up, if I copy that *.db files into this new location.

The discord has a faster way to do this with a large blockchain.db download. The other db files can not just randomly be copied around without being in sync with postgres

[rameshsraj]

Thank you @madninja , I am sorry to ask too many questions, this is the first time I am configuring and the earlier setup was configured by someone it broke ...

Can I copy only the blockchain.DB/ledger.db files, and also, I do not see any activity on the DB side, when will it start syncing up with the database ... I pinged you over the discord channel as well

[madninja]

Please don’t ping in multiple places.