Code sample to convert Pandas dataframe to KDB table

[jljs]

Hi,

I am trying to find the best practices for converting pandas dataframe back to a kdb table using pyq. Could you please provide an example or a link to the examples?

Thanks

Jeffrey

[abalkin]

For simple numeric dataframes you can do something like this:

>>> f = pd.DataFrame.from_dict({'a': [1,2], 'b':[3,4]})
>>> t = q('!', f.columns, f.values.T).flip
>>> t.show()
a b
---
1 3
2 4

In the future, we may provide an add-on that registers a generic pandas to kdb+ converter. Suggestions/contributions are welcome.

[mk1102]

Hi this is really useful thanks. Will the above conversion work with a pandas DataFrame that has a date time index and some columns that are text (not numeric)? Thanks in advance . Keen to use kdb /q once data is there …but I need to get it there first and it’s in pandas at the moment

[zak-b2c2]

Actually @abalkin this is a slow solution. Here is a much quicker which works well on large datasets. Although I'm sure you guys will find an even better one :)

>>> import numpy
>>> import pandas as pd
>>> df = pd.DataFrame.from_dict({'a': [1,2], 'b':[3,4]})
>>> t = q('!', df.columns, [numpy.array(df[x]) for x in df.columns]).flip
>>> t.show()
a b
---
1 3
2 4
>>> 

[mk1102]

How about for use cases where the dataframes are varied as per my comment above. With time series and text? Is there an efficient way to convert these to kdb tables?

[zak-b2c2]

This one works for me on all q types =< 20h

[github-actions[bot]]

Stale issue message