risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.07k stars 581 forks source link

Catch and return sink writer error `ERROR: column \“some_column\” is of type `some_pg_type` but expression is of type `character varying` to user #16687

Open kwannoel opened 6 months ago

kwannoel commented 6 months ago

Describe the bug

ERROR: column \“some_column\” is of type some_pg_type but expression is of type character varying.

 Hint: You will need to rewrite or cast the expression.
  Position: 271  Call getNextException to see other errors in the batch.
  at io.grpc.Status.asRuntimeException(Status.java:530)
  at com.risingwave.connector.JDBCSink.write(JDBCSink.java:196)
  at com.risingwave.connector.SinkWriterStreamObserver.onNext(SinkWriterStreamObserver.java:132)
  com.risingwave.connector.JniSinkWriterHandler.runJniSinkWriterThread(JniSinkWriterHandler.java:40)",

Can we propagate this error back to RW, on sink create?

Perhaps we can delay reporting the first barrier, write some records downstream first then:

  1. No error => yield first barrier, along with offsets.
  2. Error => Propagate error back.

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

kwannoel commented 6 months ago

Related https://github.com/risingwavelabs/risingwave/issues/13035

lmatz commented 5 months ago

Is this because it simply does not check whether the data types in RW and PostgreSQL mismatch or not, or there is data type incompatibility issue as we test here: https://github.com/risingwavelabs/risingwave/blob/main/integration_tests/postgres-sink/create_source.sql#L46-L69

If it does not check the mismatch, I suppose we should add such test cases to the e2e test where the XXX in statement error XXX can be tested.

StrikeW commented 5 months ago

We did not check column types with downstream table right now. We can add the check during creation of a sink.

lmatz commented 5 months ago

Got it, thanks I am going through recent incidents and just want to make sure that tests cover them