bug: Starting two query processes after the upgrade resulted in a request error

[raymondchen-byte]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Version

v1.2.539

What's Wrong?

Jun 23 21:06:54 node1 bend.sh[96243]: 06c88765-d8ea-4eca-9a9e-ea0b653ee466 2024-06-23T13:06:54.668364Z ERROR databend_query::servers::flight::v1::packets::packet_data: packet_data.rs:90 Got error code data packet: FromStdError. Code: 1001, Text = Invalid argument error: The argument fields must be consistent with the columns' schema. Use e.g. &arrow2::io::flight::default_ipc_fields(&schema.fields). Jun 23 21:06:54 node1 bend.sh[96243]: 0: databend_common_exception::exception_backtrace::capture Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/common/exception/src/exception_backtrace.rs:82:17 Jun 23 21:06:54 node1 bend.sh[96243]: 1: databend_common_exception::exception::ErrorCode::from_std_error Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/common/exception/src/exception.rs:277:24 Jun 23 21:06:54 node1 bend.sh[96243]: 2: databend_common_exception::exception_into::<impl core::convert::From for databend_common_exception::exception::ErrorCode>::from Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/common/exception/src/exception_into.rs:115:18 Jun 23 21:06:54 node1 bend.sh[96243]: 3: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/core/src/result.rs:1959:27 Jun 23 21:06:54 node1 bend.sh[96243]: databend_query::servers::flight::v1::exchange::serde::exchange_serializer::serialize_block Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/servers/flight/v1/exchange/serde/exchange_serializer.rs:226:13 Jun 23 21:06:54 node1 bend.sh[96243]: 4: <databend_query::servers::flight::v1::exchange::serde::exchange_serializer::TransformScatterExchangeSerializer as databend_common_pipeline_transforms::processors::transforms::transform::BlockMetaTransform>::transform Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/servers/flight/v1/exchange/serde/exchange_serializer.rs:194:26 Jun 23 21:06:54 node1 bend.sh[96243]: 5: <databend_common_pipeline_transforms::processors::transforms::transform::BlockMetaTransformer<B,T> as databend_common_pipeline_core::processors::processor::Processor>::process Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/pipeline/transforms/src/processors/transforms/transform.rs:306:37 Jun 23 21:06:54 node1 bend.sh[96243]: 6: <alloc::boxed::Box as databend_common_pipeline_core::processors::processor::Processor>::process Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/pipeline/core/src/processors/processor.rs:230:9 Jun 23 21:06:54 node1 bend.sh[96243]: databend_common_pipeline_core::processors::processor::ProcessorPtr::process Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/pipeline/core/src/processors/processor.rs:169:9 Jun 23 21:06:54 node1 bend.sh[96243]: 7: databend_query::pipelines::executor::executor_worker_context::ExecutorWorkerContext::execute_sync_task Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/pipelines/executor/executor_worker_context.rs:169:9 Jun 23 21:06:54 node1 bend.sh[96243]: databend_query::pipelines::executor::executor_worker_context::ExecutorWorkerContext::execute_task Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/pipelines/executor/executor_worker_context.rs:132:52 Jun 23 21:06:54 node1 bend.sh[96243]: 8: databend_query::pipelines::executor::query_pipeline_executor::QueryPipelineExecutor::execute_single_thread Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/pipelines/executor/query_pipeline_executor.rs:420:35 Jun 23 21:06:54 node1 bend.sh[96243]: 9: databend_query::pipelines::executor::query_pipeline_executor::QueryPipelineExecutor::execute_threads::{{closure}}::{{closure}} Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/pipelines/executor/query_pipeline_executor.rs:379:27 Jun 23 21:06:54 node1 bend.sh[96243]: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/core/src/panic/unwind_safe.rs:272:9 Jun 23 21:06:54 node1 bend.sh[96243]: std::panicking::try::do_call Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/panicking.rs:554:40 Jun 23 21:06:54 node1 bend.sh[96243]: std::panicking::try Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/panicking.rs:518:19 Jun 23 21:06:54 node1 bend.sh[96243]: 10: std::panic::catch_unwind Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/panic.rs:142:14 Jun 23 21:06:54 node1 bend.sh[96243]: databend_common_base::runtime::catch_unwind::catch_unwind Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/common/base/src/runtime/catch_unwind.rs:46:11 Jun 23 21:06:54 node1 bend.sh[96243]: 11: databend_query::pipelines::executor::query_pipeline_executor::QueryPipelineExecutor::execute_threads::{{closure}} Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/query/service/src/pipelines/executor/query_pipeline_executor.rs:378:34 Jun 23 21:06:54 node1 bend.sh[96243]: databend_common_base::runtime::runtime_tracker::ThreadTracker::tracking_function::{{closure}}::{{closure}} Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/common/base/src/runtime/runtime_tracker.rs:208:17 Jun 23 21:06:54 node1 bend.sh[96243]: databend_common_base::runtime::thread::Thread::named_spawn::{{closure}} Jun 23 21:06:54 node1 bend.sh[96243]: at /workspace/src/common/base/src/runtime/thread.rs:78:21 Jun 23 21:06:54 node1 bend.sh[96243]: std::sys_common::backtrace::__rust_begin_short_backtrace Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/sys_common/backtrace.rs:155:18 Jun 23 21:06:54 node1 bend.sh[96243]: 12: std::thread::Builder::spawnunchecked::{{closure}}::{{closure}} Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/thread/mod.rs:529:17 Jun 23 21:06:54 node1 bend.sh[96243]: <core::panic::unwind_safe::AssertUnwindSafe as core::ops::function::FnOnce<()>>::call_once Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/core/src/panic/unwind_safe.rs:272:9 Jun 23 21:06:54 node1 bend.sh[96243]: std::panicking::try::do_call Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/panicking.rs:554:40 Jun 23 21:06:54 node1 bend.sh[96243]: std::panicking::try Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/panicking.rs:518:19 Jun 23 21:06:54 node1 bend.sh[96243]: 13: std::panic::catch_unwind Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/panic.rs:142:14 Jun 23 21:06:54 node1 bend.sh[96243]: std::thread::Builder::spawnunchecked::{{closure}} Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/thread/mod.rs:528:30 Jun 23 21:06:54 node1 bend.sh[96243]: core::ops::function::FnOnce::call_once{{vtable.shim}} Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/core/src/ops/function.rs:250:5 Jun 23 21:06:54 node1 bend.sh[96243]: 14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/alloc/src/boxed.rs:2015:9 Jun 23 21:06:54 node1 bend.sh[96243]: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce>::call_once Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/alloc/src/boxed.rs:2015:9 Jun 23 21:06:54 node1 bend.sh[96243]: std::sys::pal::unix::thread::Thread::new::thread_start Jun 23 21:06:54 node1 bend.sh[96243]: at /rustc/8ace7ea1f7cbba7b4f031e66c54ca237a0d65de6/library/std/src/sys/pal/unix/thread.rs:108:17 Jun 23 21:06:54 node1 bend.sh[96243]: 15: start_thread Jun 23 21:06:54 node1 bend.sh[96243]: 16: clone

How to Reproduce?

My meta version is v1.2.331 and my query version is v1.2.371. After I upgraded the query node to v.1.2.359, I started only one query node and everything worked fine. When I started two query nodes, an exception occurred (replace operation). The log is shown above

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

[wubx]

Update two nodes databend-query version,use

select version()

Confirm the version