xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.io
Apache License 2.0
1.06k stars 67 forks source link

BUG: FileNotFoundError: [Errno 2] No such file or directory #758

Open simplew2011 opened 7 months ago

simplew2011 commented 7 months ago

Describe the bug

A clear and concise description of what the bug is. image

To Reproduce

To help us to reproduce this bug, please provide information below:

# download dataset
wget https://atp-modelzoo.oss-cn-hangzhou.aliyuncs.com/release/datasets/WuDaoCorpus2.0_base_sample.tgz
tar zxvf WuDaoCorpus2.0_base_sample.tgz
import xorbits.pandas as pd
import xorbits.datasets as xdatasets
from datasets import load_dataset

import xorbits
xorbits.init()

data = load_dataset("./WuDaoCorpus2.0_base_sample")
print(data)

df = pd.DataFrame(pd.DataFrame(data['train']), chunk_size=1000)
print(df.shape)
print(df.dtypes)
print(df.head())
from xorbits.experimental import dedup

res = dedup(df, col="content")
print(10*"---")
print(res)
  1. Your Python version:3.8.5
  2. The version of Xorbits you use:0.7.1
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
  5. Minimized code to reproduce the error.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Add any other context about the problem here.

simplew2011 commented 7 months ago

add a line is ok: xorbits.shutdown()

aresnow1 commented 7 months ago

So, it was raised when process exit?