Investigate Why Nodeos 4.0.5 fails

[ericpassmore]

For the following block ranges, Nodeos 4.0.5 errors out while Nodeos 5.0.0-rc3 runs to completion.

• First
  • 334493396
  • 335702522
  • snapshot-2023-10-04-16-eos-v6-0334493396.bin.zst
• Second
  • 321883745
  • 323092984
  • snapshot-2023-07-23-16-eos-v6-0321883745.bin.zst
• Third
  • 284918054
  • 285608518
  • snapshot-2022-12-21-16-eos-v6-0284918054.bin.zst
• Fourth
  • 279563785
  • 280081098
  • snapshot-2022-11-20-16-eos-v6-0279563785.bin.zst
• Fifth
  • 276455080
  • 277664215
  • snapshot-2022-11-02-16-eos-v6-0276455080.bin.zst
• Sixth
  • 274037672
  • 274728649
  • snapshot-2022-10-19-16-eos-v6-0274037672.bin.zst

[ericpassmore]

Timeouts in Nodes 4.0.5. Example from get integrity hash

END_BLOCK_ACTUAL_INTEGRITY_HASH=$(curl -s http://127.0.0.1:8888/v1/producer/get_integrity_hash | python3 ${REPLAY_CLIENT_DIR}/parse_json.py "integrity_hash")

error 2023-12-17T14:11:47.545 http-1    beast_http_session.hpp:433    handle_exception     ] fc::exception: 2 timeout_exception: deadline 2023-12-17T14:10:22.033 exceeded by 85512569us 
deadline 2023-12-17T14:10:22.033 exceeded by 85512569us 
    {"d":"2023-12-17T14:10:22.033","t":85512569}
    http-1  json.hpp:62 operator()

Traceback (most recent call last):
  File "/home/enf-replay/replay-test/replay-client/parse_json.py", line 8, in <module>
    print (json.load(sys.stdin)[sys.argv[1]])
KeyError: 'integrity_hash'