danthegoodman1
commented
1 year ago Seems like they are available, and my completion was not working?
# return pyarrow table
def to_arrowTable(res):
"""convert res to arrow table"""
# try import pyarrow and pandas, if failed, raise ImportError with suggestion
try:
import pyarrow as pa
import pandas
except ImportError as e:
print(f'ImportError: {e}')
print('Please install pyarrow and pandas via "pip install pyarrow pandas"')
raise ImportError('Failed to import pyarrow or pandas') from None
if len(res) == 0:
return pa.Table.from_batches([], schema=pa.schema([]))
return pa.RecordBatchFileReader(res.bytes()).read_all()
# return pandas dataframe
def to_df(r):
""""convert arrow table to Dataframe"""
t = to_arrowTable(r)
return t.to_pandas(use_threads=True)as well as methods for getting right from query:
def query(sql, output_format="CSV"):
lower_output_format = output_format.lower()
if lower_output_format == "dataframe":
return to_df(_chdb.query(sql, "Arrow"))
elif lower_output_format == 'arrowtable':
return to_arrowTable(_chdb.query(sql, "Arrow"))
else:
return _chdb.query(sql, output_format)
auxten
A convenience method to return a dataframe or pyarrow table from a query result would be great. DuckDB has similar features: https://duckdb.org/docs/api/python/overview.html#result-conversion
I think df and arrow are the most important ones, with arrow being the top because it's easy to get a df from an arrow table with another method call (uses more mem though)