mars-project / mars

Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.
https://mars-project.readthedocs.io
Apache License 2.0
2.71k stars 326 forks source link

Integrate paddlepaddle or Mindspore #2808

Open tianlinzx opened 2 years ago

tianlinzx commented 2 years ago

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] I want to integrate paddlepaddle and mindspore , any hint ?

Describe the solution you'd like A clear and concise description of what you want to happen. I want to integrate paddlepaddle and mindspore , any hint ? Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. I want to integrate paddlepaddle and mindspore , any hint ?

Additional context Add any other context or screenshots about the feature request here.

qinxuye commented 2 years ago

For inference, it's pretty easy, no further development required.

def inder_data(record):
    # load model from oss, obs etc
    model = load_model()
    label = model(record)

df = md.read_sql()
df.apply(infer_data, axis=1, output_type='series', dtype=object)

This will call inference function on each line of data. You can use remote API as well.

For train tasks, some development may be required, base on the situation that we have integrated tf, pytorch etc, it's workable to integrate with paddlepaddle, mindspore etc.