modelscope / data-juicer

A one-stop data processing system to make data higher-quality, juicier, and more digestible for (multimodal) LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大模型提供更高质量、更丰富、更易”消化“的数据!
Apache License 2.0
2.54k stars 161 forks source link

Efficient processing OPs for scanned images and pdf #375

Open yxdyc opened 1 month ago

yxdyc commented 1 month ago

Search before continuing 先搜索,再继续

Description 描述

There is a large amount of valuable data in the format of scanned images and PDFs. We can continuously discuss and list related processing operations to be added into DJ in this thread. Some pioneering works include MAP-NEO and PDF-Extract-Kit.

Use case 使用场景

No response

Additional 额外信息

No response

Are you willing to submit a PR for this feature? 您是否乐意为此功能提交一个 PR?

github-actions[bot] commented 3 weeks ago

This issue is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this issue will be closed in 3 day.

github-actions[bot] commented 3 weeks ago

Close this stale issue.

yxdyc commented 1 week ago

WIP by @Qirui-jiao @HYLcool @yxdyc