search

blue-yonder / turbodbc

Turbodbc is a Python module to access relational databases via the Open Database Connectivity (ODBC) interface. The module complies with the Python Database API Specification 2.0.
http://turbodbc.readthedocs.io/en/latest
MIT License
607 stars 85 forks source link

Advanced use: Arrow 'double' type as supported dtype -> is not a pyarrow dtype? #392

Closed ruben-s closed 10 months ago

ruben-s commented 10 months ago

Hi,

probably a stupid question, yet here goes:

Whilst trying to write data through turbo odbc as described in the Advanced section of the readthedocs with pyarrow, it is mentioned that only certain data types are supported.

In my database table certain columns are numeric/decimal. This is known in pyarrow, yet not supported in turboodbc:

Unsupported Arrow type for column 5 of 22 (decimal128(20, 4))
turbodbc_intern.InterfaceError: Unsupported Arrow type for column 5 of 22 (decimal128(20, 4))

Looking at https://turbodbc.readthedocs.io/en/latest/pages/advanced_usage.html - supported Apache Arrow type, and from experience :-(, using float is also not supported.

In the table it is mentioned that Double is supported.

Yet as far as I can see this does not exist in pyarrow (cfr: https://arrow.apache.org/docs/python/api/datatypes.html). It seems Arrow itself does support the type double (cfr: https://arrow.apache.org/docs/cpp/api/datatype.html#factory-functions), but this is not accessible via python?

Any help is appreciated.

Thanks.

br, Ruben

ruben-s commented 10 months ago

float32 did not seem to be supported but float64 is?

anyway, would it not be more performant if decimal can be used, as this matches more closely to numeric?

closing ticket