questdb / py-questdb-client

Python client for QuestDB InfluxDB Line Protocol
https://py-questdb-client.readthedocs.io
Apache License 2.0
49 stars 7 forks source link

Add ability for user to set `create_table` and `table_partition` when saving data to QDB #90

Open newskooler opened 4 weeks ago

newskooler commented 4 weeks ago

Hi,

I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.

I use the below code to save new and old data. If the table exists - it appends to it. If it does not, it creates it and appends to it.

import pandas as pd
from questdb.ingress import Sender

df = pd.DataFrame({
    'id': pd.Categorical(['toronto1', 'paris3']),
    'temperature': [20.0, 21.0],
    'humidity': [0.5, 0.6],
    'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})

conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    sender.dataframe(df, table_name='sensors', at='timestamp')

This causes a few issue: 1) upon creation it always creates a partition by DAY

Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency

2) It silently creates a table by default, even if one does not exist. Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.

This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.