cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.12k stars 3.81k forks source link

unexpected error when starting core changefeed with parquet format #130949

Open stevendanna opened 1 month ago

stevendanna commented 1 month ago

Describe the problem

> create table t (pk int primary key, v decimal); 
> create changefeed for t with  format = 'parquet';                                                                                                                                                                                                         
ERROR: internal error: Expected a sink with encoder for, found *changefeedccl.bufferSink
SQLSTATE: XX000
DETAIL: stack trace:
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/sink.go:497: EncodeAndEmitRow()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/event_processing.go:481: encodeForParquet()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/event_processing.go:425: encodeAndEmit()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/event_processing.go:379: ConsumeEvent()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:770: tick()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:696: Next()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_processors.go:1466: Next()
github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:198: Run()
github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:732: Run()
github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:579: Run()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928: Run()
github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_dist.go:328: func1()
github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168: func1()
golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78: func1()
src/runtime/asm_arm64.s:1222: goexit()

HINT: You have encountered an unexpected error.

Please check the public issue tracker to check whether this problem is
already tracked. If you cannot find it there, please report the error
with details by creating a new issue.

If you would rather not post publicly, please contact us directly
using the support form.

We appreciate your feedback.

Jira issue: CRDB-42304

Epic CRDB-41786

blathers-crl[bot] commented 1 month ago

Hi @stevendanna, please add branch-* labels to identify which branch(es) this C-bug affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] commented 1 month ago

cc @cockroachdb/cdc

rharding6373 commented 1 month ago

We only support parquet with [cloud] storage, so we don't expect this to work for core changefeeds. We should make sure this fails with a good error message during validation.

kaustubhbabar5 commented 1 month ago

Hii, I would like to work on this

kaustubhbabar5 commented 1 month ago

We only support parquet with [cloud] storage, so we don't expect this to work for core changefeeds. We should make sure this fails with a good error message during validation.

As parquet is only supported for cloud storage sink. I think we should not allow creating sinkless changefeed with format = 'parquet' option irrespective of whether it's core or enterprise changefeed