search

ydb-platform / ydb

YDB is an open source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions
https://ydb.tech
Apache License 2.0
4.02k stars 582 forks source link

Serialization error: batch is not valid: Invalid: Buffer #1 too small #11729

Closed fexolm closed 3 hours ago

fexolm commented 3 days ago

version: stable-24-3-13 database: olap-click-perf

Nov 19 02:16:28 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: VERIFY failed (2024-11-19T02:16:28.641775+0300): tablet_id=72075186767817822;parent_id=[50009:7438762186202536004:445];column=_yql_write_id: uint64;id=4294967042;verification=status.ok();fline=validation.cpp:7;problem=Serialization error: batch is not valid: Invalid: Buffer #1 too small in array of type uint64 and length 13358: expected at least 106864 byte(s), got 1670;
Nov 19 02:16:28 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]:   contrib/ydb/library/actors/core/log.cpp:745
Nov 19 02:16:28 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]:   ~TVerifyFormattedRecordWriter(): requirement false failed
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 0. /-S/util/system/yassert.cpp:55: NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) @ 0x55FD628F132B
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 1. /-S/contrib/ydb/library/actors/core/log.cpp:745: NActors::TVerifyFormattedRecordWriter::~TVerifyFormattedRecordWriter() @ 0x55FD62E9E167
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 2. /-S/contrib/ydb/library/formats/arrow/validation/validation.cpp:7: NKikimr::NArrow::TStatusValidator::Validate(arrow::Status const&) @ 0x55FD66BB3C88
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 3. /-S/contrib/ydb/library/formats/arrow/validation/validation.h:21: std::__y1::shared_ptr<arrow::RecordBatch> NKikimr::NArrow::TStatusValidator::GetValid<std::__y1::shared_ptr<arrow::RecordBatch>>(arrow::Result<std::__y1::shared_ptr<arrow::RecordBatch>>&&) @ 0x55FD66CA41BB
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 4. /-S/contrib/ydb/core/formats/arrow/save_load/loader.cpp:55: NKikimr::NArrow::NAccessor::TColumnLoader::ApplyVerified(TBasicString<char, std::__y1::char_traits<char>> const&, unsigned int) const @ 0x55FD66CA41BB
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 5. /-S/contrib/ydb/core/tx/columnshard/engines/portions/portion_info.cpp:728: NKikimr::NOlap::TPortionInfo::TAssembleBlobInfo::BuildRecordBatch(NKikimr::NArrow::NAccessor::TColumnLoader const&) const @ 0x55FD6E3AEC2D
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 6. /-S/contrib/ydb/core/tx/columnshard/engines/portions/portion_info.cpp:684: NKikimr::NOlap::TPortionInfo::TPreparedColumn::AssembleAccessor() const @ 0x55FD6E3AE4EA
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 7. /-S/contrib/ydb/core/tx/columnshard/engines/portions/portion_info.cpp:741: NKikimr::NOlap::TPortionInfo::TPreparedBatchData::AssembleToGeneralContainer(std::__y1::set<unsigned int, std::__y1::less<unsigned int>, std::__y1::allocator<unsigned int>> const&) const @ 0x55FD6E3B02F0
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 8. /-S/contrib/ydb/core/tx/columnshard/engines/portions/read_with_blobs.cpp:23: NKikimr::NOlap::TReadPortionInfoWithBlobs::RestoreBatch(NKikimr::NOlap::ISnapshotSchema const&, NKikimr::NOlap::ISnapshotSchema const&, std::__y1::set<unsigned int, std::__y1::less<unsigned int>, std::__y1::allocator<unsigned int>> const&) const @ 0x55FD6E3FF3A3
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 9. /-S/contrib/ydb/core/tx/columnshard/engines/changes/general_compaction.cpp:132: NKikimr::NOlap::NCompaction::TGeneralCompactColumnEngineChanges::BuildAppendedPortionsByChunks(NKikimr::NOlap::TConstructionContext&, std::__y1::vector<NKikimr::NOlap::TReadPortionInfoWithBlobs, std::__y1::allocator<NKikimr::NOlap::TReadPortionInfoWithBlobs>>&&) @ 0x55FD6E3F6A3F
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 10. /-S/contrib/ydb/core/tx/columnshard/engines/changes/general_compaction.cpp:173: NKikimr::NOlap::NCompaction::TGeneralCompactColumnEngineChanges::DoConstructBlobs(NKikimr::NOlap::TConstructionContext&) @ 0x55FD6E3F8459
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 11. /-S/contrib/ydb/core/tx/columnshard/engines/changes/abstract/abstract.cpp:23: NKikimr::NOlap::TColumnEngineChanges::ConstructBlobs(NKikimr::NOlap::TConstructionContext&) @ 0x55FD6E3E96C1
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 12. /-S/contrib/ydb/core/tx/columnshard/columnshard_impl.cpp:534: NKikimr::NColumnShard::TChangesTask::DoExecute(std::__y1::shared_ptr<NKikimr::NConveyor::ITask> const&) @ 0x55FD6E704C18
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 13. /-S/contrib/ydb/core/tx/conveyor/usage/abstract.cpp:13: NKikimr::NConveyor::ITask::Execute(std::__y1::shared_ptr<NKikimr::NConveyor::TTaskSignals>, std::__y1::shared_ptr<NKikimr::NConveyor::ITask> const&) @ 0x55FD6E37805E
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 14. /-S/contrib/ydb/core/tx/conveyor/service/worker.cpp:10: NKikimr::NConveyor::TWorker::ExecuteTask(NKikimr::NConveyor::TWorkerTask const&) @ 0x55FD75B178F5
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 15. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:251: NActors::TGenericExecutorThread::TProcessingResult NActors::TGenericExecutorThread::Execute<NActors::TMailboxTable::THTSwapMailbox>(NActors::TMailboxTable::THTSwapMailbox*, unsigned int, bool) @ 0x55FD62E7CAF0
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 16. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:440: NActors::TGenericExecutorThread::ProcessExecutorPool(NActors::IExecutorPool*)::$_0::operator()(unsigned int, bool) const @ 0x55FD62E7254A
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 17. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:493: NActors::TGenericExecutorThread::ProcessExecutorPool(NActors::IExecutorPool*) @ 0x55FD62E71D9F
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 18. /-S/contrib/ydb/library/actors/core/executor_thread.cpp:524: NActors::TExecutorThread::ThreadProc() @ 0x55FD62E73184
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 19. /-S/util/system/thread.cpp:244: (anonymous namespace)::TPosixThread::ThreadProxy(void*) @ 0x55FD628FF509
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 20. ??:0: ?? @ 0x7F882CA3F608
Nov 19 02:16:43 ydb-ru-prestable-vlx-0003 kikimr_31093[1355139]: 21. ??:0: ?? @ 0x7F882C95F132
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Key file not found!
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Determined node ID: 0
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Trying to register dynamic node to ydb-ru-prestable-klg-0028.search.yandex.net:2135
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Success. Registered via discovery service as 50009
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Node name:
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Trying to get configs from ydb-ru-prestable-vla-0009.search.yandex.net:2135
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Success.
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: configured
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Starting Kikimr r14302450 built by sandbox
Nov 19 02:16:46 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: UDFsDir is not specified, no dynamic UDFs will be loaded.
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Unknown grpc service "experimental" was not enabled!
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: Unknown grpc service "experimental" was not enabled!
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: GRpc memory quota was set but disabled due to issues with grpc quoter, to enable it use EnableGRpcMemoryQuota option
Nov 19 02:16:47 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: GRpc memory quota was set but disabled due to issues with grpc quoter, to enable it use EnableGRpcMemoryQuota option
Nov 19 02:16:50 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]: VERIFY failed (2024-11-19T02:16:50.672990+0300): tablet_id=72075186767817840;parent_id=[50009:7438762278765279122:390];column=_yql_write_id: uint64;id=4294967042;verification=status.ok();fline=validation.cpp:7;problem=Serialization error: batch is not valid: Invalid: Buffer #1 too small in array of type uint64 and length 13405: expected at least 107240 byte(s), got 1676;
Nov 19 02:16:50 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]:   contrib/ydb/library/actors/core/log.cpp:745
Nov 19 02:16:50 ydb-ru-prestable-vlx-0003 kikimr_31093[1356126]:   ~TVerifyFormattedRecordWriter(): requirement false failed
zverevgeny commented 2 days ago

Вот тут собраны все рестарты https://paste.yandex-team.ru/727adf3a-3597-49c3-a14c-39b620908aa7/text Но потом они внезапно прекратились до того, как мы смогли снять нужную диагностику Закрываю до воспроизведения

zverevgeny commented 1 day ago

Repro on updating https://nda.ya.ru/t/IstUuDv679omE5 from 24-3-12? to 24-3-13: https://paste.yandex-team.ru/991650c9-f68f-46b4-89be-d7b9601a72d2

zverevgeny commented 5 hours ago

Недоимплмент модифицируюищих транзакций. Сейчас эта функциональность отключена фиче-флагом по-умолчанию. Воспроизведение есть только на базах, где эта фича была включена