questdb / py-questdb-client

Python client for QuestDB InfluxDB Line Protocol
https://py-questdb-client.readthedocs.io
Apache License 2.0
49 stars 7 forks source link

Pandas: Support `datetime64[ns]` dataframe index as designated timestamp. #35

Open amunra opened 1 year ago

amunra commented 1 year ago

It's common to use a datetime64[ns] df.index in Pandas when dealing with timeseries. In such case our API should just be:

buffer.dataframe(df, table_name="some_name")

This means changing the default logic of the at argument to also accept two new singleton types:

buffer.dataframe(df, ..., at=Server)  # timestamps are set by the server -- the current default.
buffer.dataframe(df, ..., at=Index)  # Use the index.

The new behaviour for the at=None default would be to:

Whilst technically a breaking change, the feature change is minor and is very unlikely to affect any of our users, thus this feature will not require a new major software release number.

amunra commented 1 year ago

What is also neat is that since we can already pluck the table name off the index, we can end up in a situation where we can fully ingest a pandas dataframe with no additional args.

I.e.:

buffer.dataframe(df)

Short and sweet :-)

javier commented 1 year ago

For context, the pandas docs on indices for time-series. I see them very often specially when doing downsampling or filling gaps in your data (equivalent to questdb's FILL)

javier commented 1 year ago

Also, when using a named index, I would expect the designated timestamp column in QuestDB to retain the name. Otherwise when I have to use the column in a select statement it is confusing