Open simplew2011 opened 7 months ago
A clear and concise description of what the bug is.
To help us to reproduce this bug, please provide information below:
# download dataset wget https://atp-modelzoo.oss-cn-hangzhou.aliyuncs.com/release/datasets/WuDaoCorpus2.0_base_sample.tgz tar zxvf WuDaoCorpus2.0_base_sample.tgz
import xorbits.pandas as pd import xorbits.datasets as xdatasets from datasets import load_dataset import xorbits xorbits.init() data = load_dataset("./WuDaoCorpus2.0_base_sample") print(data) df = pd.DataFrame(pd.DataFrame(data['train']), chunk_size=1000) print(df.shape) print(df.dtypes) print(df.head()) from xorbits.experimental import dedup res = dedup(df, col="content") print(10*"---") print(res)
A clear and concise description of what you expected to happen.
Add any other context about the problem here.
add a line is ok: xorbits.shutdown()
So, it was raised when process exit?
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
To help us to reproduce this bug, please provide information below:
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
Add any other context about the problem here.