questdb / py-questdb-client

Python client for QuestDB InfluxDB Line Protocol
https://py-questdb-client.readthedocs.io
Apache License 2.0
50 stars 8 forks source link

Importing pandas dataframes does not assign timestamp correctly #47

Closed tlk3 closed 11 months ago

tlk3 commented 1 year ago

Even this sample code provided in the docs is broken:

` df = pd.DataFrame({ 'id': pd.Categorical(['toronto1', 'paris3']), 'temperature': [20.0, 21.0], 'humidity': [0.5, 0.6], 'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02']) })

with Sender('localhost', 9009) as sender: sender.dataframe(df, table_name='sensors') `

image

I've tried various things, including sending a UNIX timestamp, UNIX timestamp converted to microseconds, nanoseconds, etc.

amunra commented 1 year ago

You end up in this situation when:

To resolve, specify at='timestamp'.

See: https://py-questdb-client.readthedocs.io/en/latest/examples.html#pandas-basics

Also, here are the API docs: https://py-questdb-client.readthedocs.io/en/latest/api.html#questdb.ingress.Buffer.dataframe

Same matter, btw, if you use the .row API.

Why this happens: The Python client produces messages in the ILP protocol. In the protocol the designated timestamp is sent as a special field with nanosecond precision instead of microsecond precision.