I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.
I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.
import pandas as pd
from questdb.ingress import Sender
df = pd.DataFrame({
'id': pd.Categorical(['toronto1', 'paris3']),
'temperature': [20.0, 21.0],
'humidity': [0.5, 0.6],
'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})
conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
sender.dataframe(df, table_name='sensors', at='timestamp')
This causes a few issue:
1) upon creation it always creates a partition by DAY
Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency
2) It silently creates a table by default, even if one does not exist.
Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.
This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.
Hi,
I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.
I use the below code to save new and old data. If the table exists - it appends to it. If it does not, it creates it and appends to it.
This causes a few issue: 1) upon creation it always creates a partition by
DAY
Feature request: add a parameter which is by default
DAY
but can be changed by the user to beMONTH
,NONE
or other partition frequency2) It silently creates a table by default, even if one does not exist. Feature request: add a parameter which is by default
True
(to create a table) but can be also set toFalse
such that if a table does not exist, it will not create one.This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.