search

eosdac / eosdac-api

DAC API which takes data from state history
11 stars 11 forks source link

filler stuck at 59728747 #2

Closed ghost closed 5 years ago

ghost commented 5 years ago

Screenshot from 2019-05-26 07-20-08

tried this: pm2 delete all pm2 start CONFIG=mainnet ./eosdac-filler.js -r -s 59727000 filler still gets stuck

also re-ran CONFIG=mainnet node ./watchers/replay.js watcher seems to run through ok (though there was one error reading proposal near 59346219)

ghost commented 5 years ago

using latest commit, and eosio-statereceiver latest commit pm2 error logs not showing anything I purged the rabbitmq queues, didn't help

michaeljyeates commented 5 years ago

you shouldnt need to start the filler with -r and -s, it can be done via pm2 once things are up and running

michaeljyeates commented 5 years ago

can you post your mainnet.config.js file here please

ghost commented 5 years ago

Screenshot from 2019-05-28 00-02-44

ghost commented 5 years ago

Screenshot from 2019-05-28 00-23-32

piecesnbits commented 5 years ago

to reproduce: start filler on current block and confirm that everything works by casting a custodian vote (verify that it is caught). next propose a custodian contract config change via msig. confirm that it's not added "proposal not found"

michaeljyeates commented 5 years ago

Can you check the cpu usage while running pm2.

There seem to be 2 problems here and I cannot reproduce either, the filler hanging is different to "proposal not found".

I am going to investigate what is happening with the transaction listed because it is not a dac related msig

michaeljyeates commented 5 years ago

It would help to copy and paste the block and transaction id of the propose action which created the update config proposal, please dont send screenshots because i have to manually type it in

michaeljyeates commented 5 years ago

Once we know the block of the transaction which is failing to store we can run the following command

CONFIG=mainnet ./eosdac-filler.js -t [block_number]

This will attempt to fetch the single block, it should put it into the queue and the processor should output something

michaeljyeates commented 5 years ago

You can keep the pm2 processor running when you run the filler manually (the pm2 filler can run too), run logs on the processor with this command

pm2 logs [pm2 process number]

ghost commented 5 years ago

the following block number hangs: CONFIG=mainnet ./eosdac-filler.js -t 59728751

Screenshot from 2019-05-28 06-43-39

top shows CPU less than 1% for each of the 8 running node processes and mongod

ghost commented 5 years ago

Screenshot from 2019-05-28 08-06-09 ^ CONFIG=mainnet ./eosdac-filler.js -t 59728751 (with added console logging to eosio-statereceiver/connections.js) getting some nulls there like this_block, then statereceiver.receivedBlock does nothing and just returns. it just hangs there, no errors in the pm2 logs

statereceiver seems to process only nulls for block numbers >= 59728751

ghost commented 5 years ago

switched to a different eosdac state history node, and the filler works a charm. thanks @piecesnbits for that suggestion on telegram.