search

uccross / skyhookdm-ceph-cls

Skyhook Data Management: Storage and management of tabular data in Ceph.
https://www.skyhookdm.com
GNU Lesser General Public License v2.1
13 stars 9 forks source link

Add timestamp types #19

Closed jlefevre closed 4 years ago

jlefevre commented 5 years ago

We should add timestamp to SDT_TYPEs here. Currently we only have SDT_DATE (only mmddyy), Ashay suggests to add new type names SDT_TSp, SDT_TIME, SDT_TIMETZ.

Corresponding Postgres types are here.

We can use posix time (ptime) structs as here, which have string constructors.

Currently dates are stored as strings and extracted here as a string, from the flatbuffer format; then converted/compared here.

If the date passes the compare function, then it is re-encoded into the output (flatbuffer format) here.

And also will need a corresponding output as csv print method, and a PostgreSQL binary format.

jlefevre commented 5 years ago

Follow up: for now let's not use posix time structs, we will use the Postgres format (gregorian_date.julian_day()) 32 bit integer with offset. Also for now lets use the postgres int32 dates, then we can consider adding the postgres int64 timestamp types next as needed.

However please look into posix time structs for next steps, and keep notes here regarding if this is a better format for skyhook to use.

jlefevre commented 4 years ago

Closing this since Arrow supports these data types natively, so while moving to Arrow we will not need to convert from string types to native types.