modin-project / modin

Modin: Scale your Pandas workflows by changing a single line of code
http://modin.readthedocs.io
Apache License 2.0
9.81k stars 651 forks source link

Support partitioned parquet files #5766

Open dchigarev opened 1 year ago

dchigarev commented 1 year ago

At the time, Modin fallback to pandas on parquet files that have partitioned columns:

import pandas
import modin.pandas as pd

file_name = "modin_issue#0000.parquet"

pandas.DataFrame({"a": [1, 2, 3, 4], "b": [1, 1, 2, 2]}).to_parquet(file_name, partition_cols=["b"])

df = pd.read_parquet(file_name) # 'default to pandas' warning
print(df)
Egor-Krivov commented 1 year ago

This issue is relevant for optiver volatility benchmark https://github.com/intel-ai/omniscripts/pull/357

Liquidmasl commented 1 month ago

This would be a very important feature, it can also be used to further improve performance for sorting, grouping or filtering.

(but right now, just making read_parquet() work in generell would be great