xanderdunn / SwiftArrow

Swift wrapper of the Apache Arrow library
1 stars 0 forks source link

Datetime Conversion Imprecision #1

Open xanderdunn opened 3 years ago

xanderdunn commented 3 years ago

Be aware and potentially resolve the fact that the Date object in Swift is not perfectly represented by an Int64 in nanoseconds.

This can potentially cause several problems:

For now I will take on this imprecision but in the future I would like to fix it.

One solution to this would be to always store and manipulates Datetimes as Int64 as nanoseconds since the unix epoch. This would require changes in the BigQuery schema and how the data from FTX are converted into BigQuery data.

xanderdunn commented 3 years ago

This is more of a problem than I thought. This Date, saved to .feather and then loaded back to a Date from file, is precise only down to milliseconds:

1602034761.435837
1602034761.435742