search

segmentio / parquet-go

Go library to read/write Parquet files
https://pkg.go.dev/github.com/segmentio/parquet-go
Apache License 2.0
341 stars 102 forks source link

add support for parsing schema of converted type decimals #504

Closed cgilling closed 1 year ago

cgilling commented 1 year ago

The code here is 90% the same as the for LogicalType but gets the scale and precision from the schema element itself.

I was wondering if there was a good way to put in some automated tests around this addition, I tried looking at https://github.com/segmentio/parquet-go/pull/406 (which added the support for the Decimal LogicalType) but that didn't seem to add any tests.

The main impetus here is that I am working with files created from AWS DMS and it is creating the decimals with converted type, so when I am processing files and rewriting them (while making some modifications to other columns) the type is being dropped

cgilling commented 1 year ago

Previously I neglected to run the tests locally 😓 , I have since done that, and while there are some tests that fail locally they are also failing on the main branch as well (something to do with the ordering of text in the output that is being compared). So I think this should be ready to send through the testing workflow again