xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.readthedocs.io
Apache License 2.0
1.11k stars 67 forks source link

FEAT: Support `pd.DataFrame.from_remotes(tasks)` to create DataFrame from remote functions #572

Open qinxuye opened 1 year ago

qinxuye commented 1 year ago

Is your feature request related to a problem? Please describe

Now if users want to create a DataFrame from multiple remote functions, there is no way, we can add support for it.

Describe the solution you'd like

A clear and concise description of what you want to happen.

import pandas as pd
import xorbits.pandas as xpd
import xorbits.remote as mr

def create_data(i) -> pd.DataFrame:
    ...

remotes = [mr.spawn(create_data, args=i) for i in range(10)]
df = xpd.DataFrame.from_remotes(remotes)