Closed sschiessl-bcp closed 2 years ago
Mainnet feeder is running version 6 as well, so far no issue.
@sschiessl-bcp there should be more log messages about the error before the lines you pasted here. Can you please check?
testnet-2021-11","_type":"data","_id":"2.9.166401363","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125333,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401364","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125334,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401365","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125335,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401366","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125336,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401367","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125337,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401368","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125338,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401369","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125339,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401370","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125340,"_primary_term":1,"status":200}},{"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166401371","_version":584,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":4786125341,"_primary_term":1,"status":200}}]}
802929ms th_a elasticsearch_plugin.cpp:334 add_elasticsearch ] Error sending 10000 lines of bulk data to Elastic Search, the first lines are:
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166396372"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"account_history":{"id":"2.9.166396372","account":"1.2.25983","operation_id":"1.11.84519112","sequence":3554080,"next":"2.9.166396370"},"operation_history":{"trx_in_block":4,"op_in_trx":2,"operation_result":"[0,{}]","virtual_op":0,"op":"","op_object":{"fee":{"amount":100,"asset_id":"1.3.0"},"from":"1.2.25983","to":"1.2.25661","extensions":[],"amount_":{"amount":1,"asset_id":"1.3.0"}}},"operation_type":0,"operation_id_num":84519112,"block_data":{"block_num":51517550,"block_time":"2021-11-08T18:19:18","trx_id":"5de88b52fbb630ae9cb3e45e0d78b1d62c0daf7d"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166396373"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"account_history":{"id":"2.9.166396373","account":"1.2.25661","operation_id":"1.11.84519113","sequence":40747492,"next":"2.9.166396371"},"operation_history":{"trx_in_block":4,"op_in_trx":3,"operation_result":"[0,{}]","virtual_op":0,"op":"","op_object":{"fee":{"amount":100,"asset_id":"1.3.0"},"from":"1.2.25983","to":"1.2.25661","extensions":[],"amount_":{"amount":1,"asset_id":"1.3.0"}}},"operation_type":0,"operation_id_num":84519113,"block_data":{"block_num":51517550,"block_time":"2021-11-08T18:19:18","trx_id":"5de88b52fbb630ae9cb3e45e0d78b1d62c0daf7d"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166396374"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"account_history":{"id":"2.9.166396374","account":"1.2.25983","operation_id":"1.11.84519113","sequence":3554081,"next":"2.9.166396372"},"operation_history":{"trx_in_block":4,"op_in_trx":3,"operation_result":"[0,{}]","virtual_op":0,"op":"","op_object":{"fee":{"amount":100,"asset_id":"1.3.0"},"from":"1.2.25983","to":"1.2.25661","extensions":[],"amount_":{"amount":1,"asset_id":"1.3.0"}}},"operation_type":0,"operation_id_num":84519113,"block_data":{"block_num":51517550,"block_time":"2021-11-08T18:19:18","trx_id":"5de88b52fbb630ae9cb3e45e0d78b1d62c0daf7d"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166396375"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"account_history":{"id":"2.9.166396375","account":"1.2.25661","operation_id":"1.11.84519114","sequence":40747493,"next":"2.9.166396373"},"operation_history":{"trx_in_block":4,"op_in_trx":4,"operation_result":"[0,{}]","virtual_op":0,"op":"","op_object":{"fee":{"amount":100,"asset_id":"1.3.0"},"from":"1.2.25983","to":"1.2.25661","extensions":[],"amount_":{"amount":1,"asset_id":"1.3.0"}}},"operation_type":0,"operation_id_num":84519114,"block_data":{"block_num":51517550,"block_time":"2021-11-08T18:19:18","trx_id":"5de88b52fbb630ae9cb3e45e0d78b1d62c0daf7d"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"index":{"_index":"testnet-2021-11","_type":"data","_id":"2.9.166396376"}}
802929ms th_a elasticsearch_plugin.cpp:338 add_elasticsearch ] es.bulk_lines[i]: {"account_history":{"id":"2.9.166396376","account":"1.2.25983","operation_id":"1.11.84519114","sequence":3554082,"next":"2.9.166396374"},"operation_history":{"trx_in_block":4,"op_in_trx":4,"operation_result":"[0,{}]","virtual_op":0,"op":"","op_object":{"fee":{"amount":100,"asset_id":"1.3.0"},"from":"1.2.25983","to":"1.2.25661","extensions":[],"amount_":{"amount":1,"asset_id":"1.3.0"}}},"operation_type":0,"operation_id_num":84519114,"block_data":{"block_num":51517550,"block_time":"2021-11-08T18:19:18","trx_id":"5de88b52fbb630ae9cb3e45e0d78b1d62c0daf7d"}}
802929ms th_a elasticsearch_plugin.cpp:193 update_account_histo ] Error adding data to Elastic Search: block num 51517626, account 1.2.25661, data {"id":"1.11.84521613","op":[0,{"fee":{"amount":100,"asset_id":"1.3.0"},"from":"1.2.25983","to":"1.2.25661","amount":{"amount":1,"asset_id":"1.3.0"},"extensions":[]}],"result":[0,{}],"block_num":51517626,"trx_in_block":4,"op_in_trx":3,"virtual_op":0}
802940ms th_a db_notify.cpp:572 notify_applied_block ] Caught plugin exception: 3100000 plugin_exception: plugin exception
Error populating ES database, we are going to keep trying.
{}
th_a elasticsearch_plugin.cpp:543 operator()
802941ms th_a db_block.cpp:222 _push_block ] Failed to push new block:
3100000 plugin_exception: plugin exception
Error populating ES database, we are going to keep trying.
{}
th_a elasticsearch_plugin.cpp:543 operator()
{"next_block.block_num()":51517626}
th_a db_block.cpp:669 _apply_block
802942ms th_a db_management.cpp:227 open ] 3100000 plugin_exception: plugin exception
The log messages are still truncated. There may be an error among the 10000 lines of bulk data. BTW I have synced a testnet ES node without trouble. But I didn't enable es_objects plugin. I re-enabled it and it's replaying.
OK I caught that error on my node. Will look into it.
Here is the error message:
{"index":{"_index":"btstesthis-2021-11","_type":"data","_id":"2.9.166398662","status":400,"error":{"type":"illegal_argument_exception","reason":"mapper [operation_history.op_object.acceptable_collateral] cannot be changed from type [text] to [ObjectMapper]"}}}
FWIW the error message was found in default.log
.
It was explained here: https://github.com/bitshares/bitshares-core/pull/1396#issuecomment-432725835, In ES,
... all values in the array must be of the same datatype ...
But the data to process now is like this
"acceptable_collateral":[["1.3.1515",{"base":{"amount":10000000,"asset_id":"1.3.0"},"quote":{"amount":4000000,"asset_id":"1.3.1515"}}],["1.3.1516",{"base":{"amount":100000,"asset_id":"1.3.0"},"quote":{"amount":3000,"asset_id":"1.3.1516"}}]], "acceptable_borrowers":[["1.2.0",5000000],["1.2.6",8000000]],"extensions":[]}]],
By the way, it is strange that the error didn't pop up on the first sync, but popped up on replay. (UPDATE: see https://github.com/bitshares/bitshares-core/issues/2693 and https://github.com/bitshares/bitshares-core/issues/2493)
Thanks for the investigation. Do you have an idea how to fix it?
@sschiessl-bcp Yes, I'm working on a fix.
Actually I had anticipated this issue (see #2554) but forgot to re-check or fix.
Anyway now I expect that #2553 will fix it.
Fixed by #2553.
Bug Description ES fails to sync with bitshares/bitshares-core:test-6.0.2 and
The error is
Impacts Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.
Steps To Reproduce Steps to reproduce the behavior (example outlined below): Try syncing ES
Expected Behavior Syncing works
CORE TEAM TASK LIST