heavyai / heavydb

HeavyDB (formerly OmniSciDB)
https://heavy.ai
Apache License 2.0
2.93k stars 445 forks source link

`ArrowForeignStorage` does not support importing `date32[day]` type #678

Open dchigarev opened 2 years ago

dchigarev commented 2 years ago

Attempt to import Arrow table containing a column with date32 type results into a RuntimeError thrown from ArrowForeignStorage::getOmnisciType: https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/DataMgr/ForeignStorage/ArrowForeignStorage.cpp#L817-L818

Reproducer with pydbe ```python import sys sys.setdlopenflags(1 | 256) # RTLD_LAZY+RTLD_GLOBAL from omniscidbe import PyDbEngine import pyarrow as pa at = pa.Table.from_pydict({"a": [1, 2, 3, 4]}, schema=pa.schema({"a": pa.date32()})) server = PyDbEngine() server.importArrowTable("test_table", at) # RuntimeError: date32[day] is not yet supported ```

OmniSci seems to have a type with the similar name, so possibly the required change is just to add a case for the date type to the switch where the exception is happening https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/Shared/sqltypes.h#L53 and fix this todo https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/DataMgr/ForeignStorage/ArrowForeignStorage.cpp#L924-L928

cc @Garra1980

Garra1980 commented 2 years ago

Looks like should be something like https://github.com/omnisci/omniscidb/blob/416df923adebc6d2db1372986d961b563f94d7b7/QueryEngine/ArrowResultSetConverter.cpp#L972