man-group / ArcticDB

ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
http://arcticdb.io
Other
1.46k stars 93 forks source link

Sort and finalize staged data fails with pre-epoch dates #1781

Closed vasil-pashov closed 1 week ago

vasil-pashov commented 1 month ago

Describe the bug

When the index in any of the staged segments contains pre-epoch dates calling sort_and_finalize_staged_data fails.

Steps/Code to Reproduce

import pandas as pd
import numpy as np
import arcticdb as adb

ac = adb.Arctic("lmdb://test")
lib = ac.get_library("test", create_if_missing=True)
df = pd.DataFrame({"b": [0]}, index=pd.DatetimeIndex([pd.Timestamp(1969, 12, 31, 12, 59, 59)]))
lib.write(symbol, df, staged=True)
lib.sort_and_finalize_staged_data(symbol)

Throws:

InternalException: E_INVALID_ARGUMENT timestamp decreasing, current val=0, candidate=-39601000000000

Expected Results

Work just as append and write store pre-epoch dates.

OS, Python Version and ArcticDB Version

Python: 3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)] OS: Windows-10-10.0.22631-SP0 ArcticDB: master

Backend storage used

No response

Additional Context

No response