Can't write to a table with `DECIMAL(38,0)` field.

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake

Apache License 2.0

337 stars 71 forks source link

Can't write to a table with `DECIMAL(38,0)` field. #127

Open denizsurmeli opened 1 year ago

denizsurmeli commented 1 year ago

I have a column with type DECIMAL(38,0, but when I try to start ingestion I get the following error:

Ingest service exited with error DeltaWriteFailed { ending_offsets: "{\"0\":4999}", partition_counts: "{\"0\":5000}", source: Arrow { source: JsonError("Decimal(38, 0) type is not supported") } }

I believe this is a compatibility issue with the www.github.com/delta-io/delta-rs, a help would be appreciated. I am ready to provide more information.

Thanks in advance !

houqp commented 1 year ago

most likely a missing feature in arrow::ArrowWriter, we might just have to upgrade arrow.

rtyler commented 1 year ago

@denizsurmeli I am quite certain that this bug is fixed now with the current code in main. I will probably close this ticket when I write a regression test that covers this case

rtyler commented 6 months ago

@denizsurmeli :wave: hiya we just merged an upgrade to deltalake 0.16.5 and I'm extra certain that this bug should be fixed now, but haven't had the time to write a regression test for it. Would it be possible for you to test your workload again?