search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.06k stars 581 forks source link

Bug(Sink): Iceberg Sink picks up hidden columns #19470

Open Shreyas220 opened 1 day ago

Shreyas220 commented 1 day ago

Describe the bug

When i try to create an iceberg sink i get a schema mismatch issue can see this 

[ColumnCatalog { column_desc: Some(ColumnDesc { column_type: INT16, column_id: 3, name: "kv_log_store_row_op", version: 1 }), is_hidden: false },

Error message/log

CREATE SINK iceberg_sink_queries AS 
SELECT 
    alias,
    query_id,
    query_hash,
    start_time,
    cluster_name,
    query,
    workspace,
    cluster_uuid,
    end_time FROM consumer
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = 'true',
    s3.endpoint = 'http://minio:9000',
    s3.access.key = 'admin',
    s3.secret.key = 'password',
    s3.region = 'us-east-1',
    catalog.type = 'rest',
    catalog.name = 'demo',
    catalog.uri = 'http://rest:8181',
    warehouse.path = 's3://warehouse/',
    database.name = 'shreyas',      -- Iceberg database name
    table.name = 'queries'          -- Iceberg table name
);

ERROR:  Failed to run the query

Caused by these errors (recent errors listed first):
  1: gRPC request to meta service failed: Internal error
  2: get error from control stream, in worker node 6
  3: gRPC request to stream service failed: Internal error
  4: recv actor failure
  5: Actor 157 exited unexpectedly
  6: Executor error
  7: exchange channel to downstream actor 145 closed unexpectedly

In compute Node 

2024-11-20 08:46:41 2024-11-20T03:16:41.021627927Z  WARN risingwave_stream::task::barrier_manager: actor error overwritten actor_id=154 prev_err=Actor 154 exited unexpectedly: mismatch schema: log store: [ColumnCatalog { column_desc: Some(ColumnDesc { column_type: INT16, column_id: 3, name: "kv_log_store_row_op", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 4, name: "alias", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 5, name: "query_id", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 6, name: "query_hash", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 7, name: "start_time", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 8, name: "cluster_name", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 9, name: "query", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 10, name: "workspace", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 11, name: "cluster_uuid", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 12, name: "end_time", version: 1 }), is_hidden: false }], input: Schema { fields: [consumer_table.alias:Varchar, consumer_table.query_id:Varchar, consumer_table.query_hash:Varchar, consumer_table.start_time:Varchar, consumer_table.cluster_name:Varchar, consumer_table.query:Varchar, consumer_table.workspace:Varchar, consumer_table.cluster_uuid:Varchar, consumer_table.end_time:Varchar, $expr1:Struct(StructType { field_names: ["cluster_uuid"], field_types: [Varchar] })] }

To Reproduce

CREATE SOURCE consumer (
    alias varchar ,
    query_id bigint ,
    query_hash varchar,
    start_time varchar,
    status varchar,
    cluster_name varchar,
    query varchar,
    workspace varchar,
    cluster_uuid varchar,
    end_time bigint
)
WITH (
    connector = 'mqtt',
    topic = 'queries',
    url = 'url',
    qos = 'at_least_once',
    username = '',
    password = ''
) FORMAT PLAIN ENCODE JSON;
 CREATE SINK iceberg_sink_queries AS 
SELECT 
    alias,
    query_id,
    query_hash,
    start_time,
    cluster_name,
    query,
    workspace,
    cluster_uuid,
    end_time FROM consumer
WITH (
    connector = 'iceberg',
    type = 'append-only',
    force_append_only = 'true',
    s3.endpoint = 'http://minio:9000',
    s3.access.key = 'admin',
    s3.secret.key = 'password',
    s3.region = 'us-east-1',
    catalog.type = 'rest',
    catalog.name = 'demo',
    catalog.uri = 'http://rest:8181',
    warehouse.path = 's3://warehouse/',
    database.name = 's1',     
    table.name = 't1'          
);

schema in iceberg is 

spark-sql ()> describe demo.s1.t1;
alias                   string                                      
query_id                string                                      
query_hash              string                                      
start_time              string                                      
cluster_name            string                                      
query                   string                                      
workspace               string                                      
cluster_uuid            string                                      
end_time                string                                      
# Partition Information                                             
# col_name              data_type               comment             
cluster_uuid            string                                      
Time taken: 0.255 seconds, Fetched 12 row(s)

Expected behavior

I should be able to create an iceberg sink

How did you deploy RisingWave?

docker-compose-distributed.yml

The version of RisingWave

select version();
                                       version                                       
-------------------------------------------------------------------------------------
 PostgreSQL 13.14.0-RisingWave-2.1.0-rc.1 (a7d8b79232c8395e54839803c09f83b4771125e3)
(1 row)

Additional context

No response

Shreyas220 commented 1 day ago

Backtrace


2024-11-20 08:46:38 2024-11-20T03:16:38.019500967Z ERROR risingwave_stream::task::stream_manager: actor exit with error actor_id=147 error=mismatch schema: log store: [ColumnCatalog { column_desc: Some(ColumnDesc { column_type: INT16, column_id: 3, name: "kv_log_store_row_op", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 4, name: "alias", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 5, name: "query_id", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 6, name: "query_hash", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 7, name: "start_time", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 8, name: "cluster_name", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 9, name: "query", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 10, name: "workspace", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 11, name: "cluster_uuid", version: 1 }), is_hidden: false }, ColumnCatalog { column_desc: Some(ColumnDesc { column_type: VARCHAR, column_id: 12, name: "end_time", version: 1 }), is_hidden: false }], input: Schema { fields: [consumer_table.alias:Varchar, consumer_table.query_id:Varchar, consumer_table.query_hash:Varchar, consumer_table.start_time:Varchar, consumer_table.cluster_name:Varchar, consumer_table.query:Varchar, consumer_table.workspace:Varchar, consumer_table.cluster_uuid:Varchar, consumer_table.end_time:Varchar, $expr1:Struct(StructType { field_names: ["cluster_uuid"], field_types: [Varchar] })] }
2024-11-20 08:46:38 
2024-11-20 08:46:38 Backtrace:
2024-11-20 08:46:38    0: std::backtrace_rs::backtrace::libunwind::trace
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
2024-11-20 08:46:38    1: std::backtrace_rs::backtrace::trace_unsynchronized
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
2024-11-20 08:46:38    2: std::backtrace::Backtrace::create
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/backtrace.rs:331:13
2024-11-20 08:46:38    3: anyhow::error::<impl anyhow::Error>::msg
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.86/src/backtrace.rs:27:14
2024-11-20 08:46:38    4: risingwave_stream::from_proto::sink::resolve_pk_info
2024-11-20 08:46:38    5: <risingwave_stream::from_proto::sink::SinkExecutorBuilder as risingwave_stream::from_proto::ExecutorBuilder>::new_boxed_executor::{{closure}}
2024-11-20 08:46:38              at ./risingwave/src/stream/src/from_proto/sink.rs:267:31
2024-11-20 08:46:38    6: risingwave_stream::from_proto::create_executor::{{closure}}
2024-11-20 08:46:38              at ./risingwave/src/stream/src/from_proto/mod.rs:128:77
2024-11-20 08:46:38    7: risingwave_stream::task::stream_manager::<impl risingwave_stream::task::barrier_manager::StreamActorManager>::create_nodes_inner::{{closure}}
2024-11-20 08:46:38              at ./risingwave/src/stream/src/task/stream_manager.rs:519:70
2024-11-20 08:46:38    8: <core::pin::Pin<P> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/future/future.rs:123:9
2024-11-20 08:46:38    9: risingwave_stream::task::stream_manager::<impl risingwave_stream::task::barrier_manager::StreamActorManager>::create_nodes::{{closure}}
2024-11-20 08:46:38              at ./risingwave/src/stream/src/task/stream_manager.rs:569:14
2024-11-20 08:46:38   10: risingwave_stream::task::stream_manager::<impl risingwave_stream::task::barrier_manager::StreamActorManager>::create_actor::{{closure}}
2024-11-20 08:46:38              at ./risingwave/src/stream/src/task/stream_manager.rs:607:18
2024-11-20 08:46:38   11: <core::pin::Pin<P> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/future/future.rs:123:9
2024-11-20 08:46:38   12: <F as futures_core::future::TryFuture>::try_poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
2024-11-20 08:46:38   13: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
2024-11-20 08:46:38   14: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
2024-11-20 08:46:38   15: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
2024-11-20 08:46:38   16: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
2024-11-20 08:46:38   17: <F as futures_core::future::TryFuture>::try_poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
2024-11-20 08:46:38   18: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/try_flatten.rs:49:61
2024-11-20 08:46:38   19: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
2024-11-20 08:46:38   20: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
2024-11-20 08:46:38   21: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
2024-11-20 08:46:38   22: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
2024-11-20 08:46:38   23: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/task_local.rs:391:31
2024-11-20 08:46:38   24: tokio::task::task_local::LocalKey<T>::scope_inner
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/task_local.rs:217:19
2024-11-20 08:46:38   25: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/task_local.rs:387:19
2024-11-20 08:46:38   26: await_tree::root::TreeRoot::instrument::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/await-tree-0.2.1/src/root.rs:43:34
2024-11-20 08:46:38   27: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
2024-11-20 08:46:38   28: core::ops::function::FnOnce::call_once
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/ops/function.rs:250:5
2024-11-20 08:46:38   29: tokio_metrics::task::instrument_poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.1/src/task.rs:2530:15
2024-11-20 08:46:38   30: <tokio_metrics::task::Instrumented<T> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-metrics-0.3.1/src/task.rs:2430:9
2024-11-20 08:46:38   31: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/task_local.rs:391:31
2024-11-20 08:46:38   32: tokio::task::task_local::LocalKey<T>::scope_inner
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/task_local.rs:217:19
2024-11-20 08:46:38   33: <tokio::task::task_local::TaskLocalFuture<T,F> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/task_local.rs:387:19
2024-11-20 08:46:38   34: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
2024-11-20 08:46:38   35: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:328:17
2024-11-20 08:46:38   36: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/loom/std/unsafe_cell.rs:16:9
2024-11-20 08:46:38   37: tokio::runtime::task::core::Core<T,S>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:317:30
2024-11-20 08:46:38   38: tokio::runtime::task::harness::poll_future::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:485:19
2024-11-20 08:46:38   39: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/panic/unwind_safe.rs:272:9
2024-11-20 08:46:38   40: std::panicking::try::do_call
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:557:40
2024-11-20 08:46:38   41: std::panicking::try
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:521:19
2024-11-20 08:46:38   42: std::panic::catch_unwind
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panic.rs:350:14
2024-11-20 08:46:38   43: tokio::runtime::task::harness::poll_future
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:473:18
2024-11-20 08:46:38   44: tokio::runtime::task::harness::Harness<T,S>::poll_inner
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:208:27
2024-11-20 08:46:38   45: tokio::runtime::task::harness::Harness<T,S>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:153:15
2024-11-20 08:46:38   46: tokio::runtime::task::raw::RawTask::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/raw.rs:201:18
2024-11-20 08:46:38   47: tokio::runtime::task::LocalNotified<S>::run
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/mod.rs:427:9
2024-11-20 08:46:38   48: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:585:18
2024-11-20 08:46:38   49: tokio::runtime::coop::with_budget
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:107:5
2024-11-20 08:46:38   50: tokio::runtime::coop::budget
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/coop.rs:73:5
2024-11-20 08:46:38   51: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:584:9
2024-11-20 08:46:38   52: tokio::runtime::scheduler::multi_thread::worker::Context::run
2024-11-20 08:46:38   53: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:500:21
2024-11-20 08:46:38   54: tokio::runtime::context::scoped::Scoped<T>::set
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/scoped.rs:40:9
2024-11-20 08:46:38   55: tokio::runtime::context::set_scheduler::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context.rs:180:26
2024-11-20 08:46:38   56: std::thread::local::LocalKey<T>::try_with
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/thread/local.rs:283:12
2024-11-20 08:46:38   57: std::thread::local::LocalKey<T>::with
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/thread/local.rs:260:9
2024-11-20 08:46:38   58: tokio::runtime::context::set_scheduler
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context.rs:180:17
2024-11-20 08:46:38   59: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:495:9
2024-11-20 08:46:38   60: tokio::runtime::context::runtime::enter_runtime
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/context/runtime.rs:65:16
2024-11-20 08:46:38   61: tokio::runtime::scheduler::multi_thread::worker::run
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:487:5
2024-11-20 08:46:38   62: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/multi_thread/worker.rs:455:45
2024-11-20 08:46:38   63: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/task.rs:42:21
2024-11-20 08:46:38   64: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
2024-11-20 08:46:38   65: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:328:17
2024-11-20 08:46:38   66: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/loom/std/unsafe_cell.rs:16:9
2024-11-20 08:46:38   67: tokio::runtime::task::core::Core<T,S>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/core.rs:317:30
2024-11-20 08:46:38   68: tokio::runtime::task::harness::poll_future::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:485:19
2024-11-20 08:46:38   69: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/panic/unwind_safe.rs:272:9
2024-11-20 08:46:38   70: std::panicking::try::do_call
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:557:40
2024-11-20 08:46:38   71: std::panicking::try
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:521:19
2024-11-20 08:46:38   72: std::panic::catch_unwind
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panic.rs:350:14
2024-11-20 08:46:38   73: tokio::runtime::task::harness::poll_future
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:473:18
2024-11-20 08:46:38   74: tokio::runtime::task::harness::Harness<T,S>::poll_inner
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:208:27
2024-11-20 08:46:38   75: tokio::runtime::task::harness::Harness<T,S>::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/harness.rs:153:15
2024-11-20 08:46:38   76: tokio::runtime::task::raw::RawTask::poll
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/raw.rs:201:18
2024-11-20 08:46:38   77: tokio::runtime::task::UnownedTask<S>::run
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/task/mod.rs:464:9
2024-11-20 08:46:38   78: tokio::runtime::blocking::pool::Task::run
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:159:9
2024-11-20 08:46:38   79: tokio::runtime::blocking::pool::Inner::run
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:513:17
2024-11-20 08:46:38   80: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
2024-11-20 08:46:38              at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:471:13
2024-11-20 08:46:38   81: std::sys::backtrace::__rust_begin_short_backtrace
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/backtrace.rs:152:18
2024-11-20 08:46:38   82: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/thread/mod.rs:542:17
2024-11-20 08:46:38   83: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/panic/unwind_safe.rs:272:9
2024-11-20 08:46:38   84: std::panicking::try::do_call
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:557:40
2024-11-20 08:46:38   85: std::panicking::try
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panicking.rs:521:19
2024-11-20 08:46:38   86: std::panic::catch_unwind
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/panic.rs:350:14
2024-11-20 08:46:38   87: std::thread::Builder::spawn_unchecked_::{{closure}}
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/thread/mod.rs:541:30
2024-11-20 08:46:38   88: core::ops::function::FnOnce::call_once{{vtable.shim}}
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/core/src/ops/function.rs:250:5
2024-11-20 08:46:38   89: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/alloc/src/boxed.rs:2070:9
2024-11-20 08:46:38   90: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/alloc/src/boxed.rs:2070:9
2024-11-20 08:46:38   91: std::sys::pal::unix::thread::Thread::new::thread_start
2024-11-20 08:46:38              at ./rustc/5affbb17153bc69a9d5d8d2faa4e399a014a211e/library/std/src/sys/pal/unix/thread.rs:108:17
2024-11-20 08:46:38   92: start_thread
2024-11-20 08:46:38              at ./nptl/pthread_create.c:447:8
2024-11-20 08:46:38   93: thread_start
2024-11-20 08:46:38              at ./misc/../sysdeps/unix/sysv/linux/aarch64/clone3.S:76
2024-11-20 08:46:38 
2024-11-20 08:46:38 
2024-11-20 08:46:41 2024-11-20T03:16:41.020989802Z  WARN risingwave_stream::task::barrier_manager: control stream reset with error error=gRPC request failed: Internal error: recv acto
chenzl25 commented 1 day ago

This issue can be resolved after removing the partition from iceberg table.