[Feat] Automatically Handle `BrokenPipeError` Caused by Limited Memory

[yxdyc]

Search before continuing 先搜索，再继续

• [X] I have searched the Data-Juicer issues and found no similar feature requests. 我已经搜索了 Data-Juicer 的 issue 列表但是没有发现类似的功能需求。

Description 描述

The hardware resources (especially memory) available at runtime can vary with different data recipes and datasets, potentially leading to BrokenPipeError in DJ's multiprocessing mode when resources are limited.

Ideally, we can automatically track the available resources and assist users in splitting the dataset into smaller subsets, allowing re-processing in batch mode.

Use case 使用场景

No response

Additional 额外信息

No response

Are you willing to submit a PR for this feature? 您是否乐意为此功能提交一个 PR？

• [x] Yes I'd like to help by submitting a PR! 是的！我愿意提供帮助并提交一个PR！

[github-actions[bot]]

This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.

[github-actions[bot]]

Close this stale issue.

[yxdyc]

Update: We are currently enhancing the Data-Juicer processing engine to automatically divide data into smaller batches. This improvement aims to boost efficiency and significantly reduce the likelihood of encountering out-of-memory (OOM) issues.