jeppe742 / DeltaLakeReader

Read Delta tables without any Spark
Apache License 2.0
47 stars 14 forks source link

Adding decimal support #31

Closed Caedin closed 3 years ago

Caedin commented 3 years ago

Adds a check for decimal types in the schema mapping. Attempts to map deltalake decimal types to pyarrow decimal types using a basic regex, and defaults to existing incorrect type message if no match is found.

Caedin commented 3 years ago

Testing

Setting up the table:

image

Insert some dummy data:

image

Sample of the output with the decimal type: image

Trying to query the data using the delta reader 0.2.8 results in the following error: image

After the fix, the query is successful, with the decimal types returning properly: image

jeppe742 commented 3 years ago

Will just merge it using my own branch, otherwise the tests will not work :)