replay fails with state_history_plugin enabled

[HuweyII]

I've tried this with on both Ubuntu 16.04LTS and 18.04LTS. Each time with a fresh block directory downloaded fresh. Each time state, state-history, and snapshots directories where deleted before the replay was started.

The only startup options where, --disable-replay-opts --data-dir --config-dir --replay

Log excerpt follows:

info  2019-02-24T08:18:31.285 thread-0  controller.cpp:313            replay               ] 14640000 of 44074497
info  2019-02-24T08:18:33.062 thread-0  controller.cpp:313            replay               ] 14640500 of 44074497
info  2019-02-24T08:18:34.694 thread-0  controller.cpp:313            replay               ] 14641000 of 44074497
info  2019-02-24T08:18:36.449 thread-0  controller.cpp:313            replay               ] 14641500 of 44074497
info  2019-02-24T08:18:39.833 thread-0  controller.cpp:313            replay               ] 14642000 of 44074497
warn  2019-02-24T08:18:41.596 thread-0  controller.cpp:236            emit                 ] signal handler threw exception
warn  2019-02-24T08:18:41.600 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missing trace for transaction 3c761209f4e57990dcfc133a3fbc6552241cc153ba85abe1864cfa87f06977c2
    {"id":"3c761209f4e57990dcfc133a3fbc6552241cc153ba85abe1864cfa87f06977c2"}
    thread-0  state_history_plugin.cpp:385 store_traces
warn  2019-02-24T08:18:41.604 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log
    {"name":"trace_history"}
    thread-0  state_history_log.hpp:84 write_entry
warn  2019-02-24T08:18:41.606 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log
    {"name":"trace_history"}
    thread-0  state_history_log.hpp:84 write_entry
warn  2019-02-24T08:18:41.610 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log

[HuweyII]

Oh, I also tried this on 1.6.2 and 1.7.0-RC1.

[HuweyII]

Fresh startup looks like this. Replay never seems to go any further.

APPBASE: Warning: The following configuration items in the config.ini file are redundantly set to
         their default value:
             bnet-endpoint, bnet-no-trx, blocks-dir, reversible-blocks-db-size-mb, contracts-console, 
             https-client-validate-peers, access-control-allow-credentials, p2p-max-nodes-per-host, 
             allowed-connection, max-clients, connection-cleanup-period, sync-fetch-span, 
             enable-stale-production, pause-on-startup, max-transaction-time, max-irreversible-block-age, 
             txn-reference-block-lag
         Explicit values will override future changes to application defaults. Consider commenting out or
         removing these items.
info  2019-02-24T22:10:05.812 thread-0  chain_plugin.cpp:333          plugin_initialize    ] initializing chain plugin
info  2019-02-24T22:10:05.816 thread-0  block_log.cpp:134             open                 ] Log is nonempty
info  2019-02-24T22:10:05.817 thread-0  block_log.cpp:161             open                 ] Index is nonempty
info  2019-02-24T22:10:05.992 thread-0  http_plugin.cpp:452           plugin_initialize    ] configured http to listen on 0.0.0.0:8888
warn  2019-02-24T22:10:05.992 thread-0  net_api_plugin.cpp:96         plugin_initialize    ] 
**********SECURITY WARNING**********
*                                  *
* --         Net API            -- *
* - EXPOSED to the LOCAL NETWORK - *
* - USE ONLY ON SECURE NETWORKS! - *
*                                  *
************************************

info  2019-02-24T22:10:05.992 thread-0  state_history_plugin.c:544    plugin_initialize    ] ip_port: 0.0.0.0:8080 host: 0.0.0.0 port: 8080 
info  2019-02-24T22:10:05.993 thread-0  state_history_log.hpp:201     open_log             ] trace_history.log has blocks 2-14642454
info  2019-02-24T22:10:05.993 thread-0  state_history_log.hpp:201     open_log             ] chain_state_history.log has blocks 2-14642454
info  2019-02-24T22:10:05.993 thread-0  main.cpp:99                   main                 ] nodeos version v1.7.0-rc1
info  2019-02-24T22:10:05.993 thread-0  main.cpp:100                  main                 ] eosio root is /home/eos/.local/share
info  2019-02-24T22:10:05.993 thread-0  main.cpp:101                  main                 ] nodeos using configuration file /export/vol01/fn/config.ini
info  2019-02-24T22:10:05.993 thread-0  main.cpp:102                  main                 ] nodeos data directory is /export/vol01/fn
info  2019-02-24T22:10:05.994 thread-0  controller.cpp:307            replay               ] existing block log, attempting to replay from 22238001 to 44074497 blocks
warn  2019-02-24T22:10:05.995 thread-0  transaction_context.cp:108    deadline_timer       ] Using polled checktime; deadline timer too inaccurate: min:5us max:3510us mean:618us stddev:321us
warn  2019-02-24T22:10:06.481 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log
    {"name":"trace_history"}
    thread-0  state_history_log.hpp:84 write_entry
warn  2019-02-24T22:10:06.732 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log
    {"name":"trace_history"}
    thread-0  state_history_log.hpp:84 write_entry
warn  2019-02-24T22:10:06.824 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception

[mozgoo]

Same problem, im use state_history_plugin and get

warn  2019-03-01T13:39:50.948 thread-0  controller.cpp:244            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log
    {"name":"trace_history"}
    thread-0  state_history_log.hpp:84 write_entry

Trying to do --replay-blockchain did not help:

warn  2019-03-02T05:16:02.532 thread-0  controller.cpp:234            emit                 ] 3110000 plugin_exception: Plugin exception
missed a block in trace_history.log
    {"name":"trace_history"}
    thread-0  state_history_log.hpp:84 write_entry

Im also use EOSIO/fill-postgresql and last block in log is

info  2019-03-01T13:09:44.416 thread-0  fill_postgresql_plugin:1148   receive_result       ] block 14642453

My nodeos config file: config.txt

[jackami]

Same problem too。 May I ask which version of this bug can be fixed? @andriantolie

TKS

[mozgoo]

v1.5.1 sync from scratch ws request to state_history_plugin method get_status_result_v0

{ head:
  { block_num: 17973673,
    block_id: '011241A9DC5506D29C88CF2C3F9A2AAD4CE9AEF9230E108F621F153DC7BB21BC' },
 last_irreversible:
  { block_num: 17973347,
    block_id: '011240632CC7D65C70F4217A1470507BC83B9836071C3BF28F7B8382D627FA54' },
 trace_begin_block: 2,
 trace_end_block: 14642455,
 chain_state_begin_block: 2,
 chain_state_end_block: 14642455 }

v1.5.6 also from scratch get_status_result_v0

{ head:
  { block_num: 14819971,
    block_id: '00E22283B04FCA5379E430D7B9948EBDFB1DC8E8016F1BFC2C0BEF62041403F6' },
 last_irreversible:
  { block_num: 14819644,
    block_id: '00E2213CF087CF5EA49BBAA18D5B96808470CCF03F79E428A6EFCC542E567E8D' },
 trace_begin_block: 2,
 trace_end_block: 14642455,
 chain_state_begin_block: 2,
 chain_state_end_block: 14642455 }

[salisbury-espinosa]

@jackami @HuweyII https://github.com/EOSIO/eos/issues/6377#issuecomment-477590397 you need to remove the history_plugin from the config

[spoonincode]

@HuweyII @mozgoo @jackami is this still a problem? Did removing history_plugin resolve the issue?

[HuweyII]

This no longer seems to be an issue.