vaexio / vaex

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
https://vaex.io
MIT License
8.23k stars 590 forks source link

[FEATURE-REQUEST]vaex concat: Add a hierarchical index at the outermost level of the data. #2285

Open oan-Dev05 opened 1 year ago

oan-Dev05 commented 1 year ago

Vaex concat with keys In pandas for concatenation along with keys is done like this: df = pd.concat([df1, df1], keys=[Dataframe 1', 'Dataframe 2']) So, how we can achieve this in vaex. help me out. Thanks!

JovanVeljanoski commented 1 year ago

You can use join in vaex.

oan-Dev05 commented 1 year ago

@JovanVeljanoski thanks

oan-Dev05 commented 1 year ago

@JovanVeljanoski I meant: axis=0 When concatenating Series objects along axis=0 (extending the index). Those keys, become a new initial level of a MultiIndex object in the index attribute. Example: pandas: pd.concat([df1, df2, df3], keys=['A', 'B', 'C'])


table column1 column 2 A 2 1 A 3 2 B 1 3 B 2 4 B 1 5 C 3 6 dtype: int64 ,

JovanVeljanoski commented 1 year ago

then you can use vaex.concat. Vaex has no concept of indices (so far).

oan-Dev05 commented 1 year ago

thanks

maartenbreddels commented 1 year ago

I think the equivalent in vaex would be to manually add a column with the same name to each dataframe:

df['identifier'] = vaex.vconstant('A')

etc