Timestamps before 1970 should be allowed for columns which are not the designated timestamp when inserting dataframes over ILP

rueschm commented 1 year ago

When inserting a dataframe with questdb.ingress.sender.dataframe I get the following error, if a timestamp column holds values which are older than 1.1.1970:

questdb.ingress.IngressError: Failed to serialize value of column 'xxxxx' at row index 346 (Timestamp('1962-01-05 00:00:00')): Timestamp -252115200000000 is negative. It must be >= 0. [dc=501007]

I am aware that this is supposed to happen if it would be the designated timestamp column (due to the known limitation of questdb), but questdb would accept older timestamps for timestamp columns other than the designated timestamp. -> The ingress library throws an error in both cases

Workaround:

create the table manually and define the column which holds the timestamp before 1970 as type timestamp
convert the timestamp column in pandas to unix time in ms

unfortunately this limitation makes it necessary to create the table manually and it can not be auto created by the library.

amunra commented 1 year ago

Hi @rueschm,

Thanks for the bug report. I'll take a look as soon as I'm back from holiday.

I'll provide an update in about a week and a half.

amunra commented 1 year ago

Fix incorporated with the upgrade to the latest c-questdb-client core implementation as part of https://github.com/questdb/py-questdb-client/pull/53.

The fix will be made available in the next release.

questdb / py-questdb-client

Timestamps before 1970 should be allowed for columns which are not the designated timestamp when inserting dataframes over ILP #34