[中文主页] | [Docs] | [API] | [DJ-SORA] | [Awesome List]
Data-Juicer is a one-stop multimodal data processing system to make data higher-quality, juicier, and more digestible for LLMs.
We provide a playground with a managed JupyterLab. Try Data-Juicer straight away in your browser! If you find Data-Juicer useful for your research or development, please kindly cite our work.
Platform for AI of Alibaba Cloud (PAI) has cited our work and integrated Data-Juicer into its data processing products. PAI is an AI Native large model and AIGC engineering platform that provides dataset management, computing power management, model tool chain, model development, model training, model deployment, and AI asset management. For documentation on data processing, please refer to: PAI-Data Processing for Large Models.
Data-Juicer is being actively updated and maintained. We will periodically enhance and add more features, data recipes and datasets. We welcome you to join us (via issues, PRs, Slack channel, DingDing group, ...), in promoting data-model co-development along with research and applications of (multimodal) LLMs!
Systematic & Reusable: Empowering users with a systematic library of 80+ core OPs, 20+ reusable config recipes, and 20+ feature-rich dedicated toolkits, designed to function independently of specific multimodal LLM datasets and processing pipelines.
Data-in-the-loop & Sandbox: Supporting one-stop data-model collaborative development, enabling rapid iteration through the sandbox laboratory, and providing features such as feedback loops based on data and model, visualization, and multidimensional automatic evaluation, so that you can better understand and improve your data and models.
Towards production environment: Providing efficient and parallel data processing pipelines (Aliyun-PAI\Ray\Slurm\CUDA\OP Fusion) requiring less memory and CPU usage, optimized with automatic fault-toleration.
Comprehensive Data Processing Recipes: Offering tens of pre-built data processing recipes for pre-training, fine-tuning, en, zh, and more scenarios. Validated on reference LLaMA and LLaVA models.
Flexible & Extensible: Accommodating most types of data formats (e.g., jsonl, parquet, csv, ...) and allowing flexible combinations of OPs. Feel free to implement your own OPs for customizable data processing.
User-Friendly Experience: Designed for simplicity, with comprehensive documentation, easy start guides and demo configs, and intuitive configuration with simple adding/removing OPs from existing configs.
Run the following commands to install the latest basic data_juicer
version in
editable mode:
cd <path_to_data_juicer>
pip install -v -e .
Some OPs rely on some other too large or low-platform-compatibility third-party libraries. You can install optional dependencies as needed:
cd <path_to_data_juicer>
pip install -v -e . # install a minimal dependencies, which support the basic functions
pip install -v -e .[tools] # install a subset of tools dependencies
The dependency options are listed below:
Tag | Description |
---|---|
. or .[mini] |
Install minimal dependencies for basic Data-Juicer. |
.[all] |
Install all dependencies except sandbox. |
.[sci] |
Install all dependencies for all OPs. |
.[dist] |
Install dependencies for distributed data processing. (Experimental) |
.[dev] |
Install dependencies for developing the package as contributors. |
.[tools] |
Install dependencies for dedicated tools, such as quality classifiers. |
.[sandbox] |
Install all dependencies for sandbox. |
data_juicer
using pip
:pip install py-data-juicer
data_juicer
and two basic tools
(data processing and analysis) are available in this way. If you want customizable
and complete functions, we recommend you install data_juicer
from source.data_juicer
, we recommend you install from source.You can
either pull our pre-built image from DockerHub:
docker pull datajuicer/data-juicer:<version_tag>
or run the following command to build the docker image including the
latest data-juicer
with provided Dockerfile:
docker build -t datajuicer/data-juicer:<version_tag> .
<version_tag>
is like v0.2.0
, which is the same as release version tag.import data_juicer as dj
print(dj.__version__)
Before using video-related operators, FFmpeg should be installed and accessible via the $PATH environment variable.
You can install FFmpeg using package managers(e.g. sudo apt install ffmpeg on Debian/Ubuntu, brew install ffmpeg on OS X) or visit the official ffmpeg link.
Check if your environment path is set correctly by running the ffmpeg command from the terminal.
process_data.py
tool or dj-process
command line tool with your config as the argument to process
your dataset.# only for installation from source
python tools/process_data.py --config configs/demo/process.yaml
# use command line tool
dj-process --config configs/demo/process.yaml
Note: For some operators that involve third-party models or resources which are not stored locally on your computer, it might be slow for the first running because these ops need to download corresponding resources into a directory first.
The default download cache directory is ~/.cache/data_juicer
. Change the cache location by setting the shell environment variable, DATA_JUICER_CACHE_HOME
to another directory, and you can also change DATA_JUICER_MODELS_CACHE
or DATA_JUICER_ASSETS_CACHE
in the same way:
Note: When using operators with third-party models, it's necessary to declare the corresponding mem_required
in the configuration file (you can refer to the settings in the config_all.yaml
file). During runtime, Data-Juicer will control the number of processes based on memory availability and the memory requirements of the operator models to achieve better data processing efficiency. When running with CUDA environment, if the mem_required for an operator is not declared correctly, it could potentially lead to a CUDA Out of Memory issue.
# cache home
export DATA_JUICER_CACHE_HOME="/path/to/another/directory"
# cache models
export DATA_JUICER_MODELS_CACHE="/path/to/another/directory/models"
# cache assets
export DATA_JUICER_ASSETS_CACHE="/path/to/another/directory/assets"
We provide various simple interfaces for users to choose from as follows.
#... init op & dataset ...
# Chain call style, support single operator or operator list
dataset = dataset.process(op)
dataset = dataset.process([op1, op2])
# Functional programming style for quick integration or script prototype iteration
dataset = op(dataset)
dataset = op.run(dataset)
We have now implemented multi-machine distributed data processing based on RAY. The corresponding demos can be run using the following commands:
# Run text data processing
python tools/process_data.py --config ./demos/process_on_ray/configs/demo.yaml
# Run video data processing
python tools/process_data.py --config ./demos/process_video_on_ray/configs/demo.yaml
ray
, e.g. ray_video_deduplicator
and ray_document_deduplicator
. Those operators also rely on a Redis instance. So in addition to starting the RAY cluster, you also need to setup your Redis instance in advance and provide host
and port
of your Redis instance in configuration.Users can also opt not to use RAY and instead split the dataset to run on a cluster with Slurm / Aliyun PAI-DLC. In this case, please use the default Data-Juicer without RAY.
analyze_data.py
tool or dj-analyze
command line tool with your config as the argument to analyze your dataset.# only for installation from source
python tools/analyze_data.py --config configs/demo/analyzer.yaml
# use command line tool
dj-analyze --config configs/demo/analyzer.yaml
app.py
tool to visualize your dataset in your browser.streamlit run app.py
config_all.yaml
which includes all ops and default
arguments. You just need to remove ops that you won't use and refine
some arguments of ops.config_all.yaml
, op documents, and advanced Build-Up Guide for developers.python xxx.py --config configs/demo/process.yaml --language_id_score_filter.lang=en
The basic config format and definition is shown below.
The data sandbox laboratory (DJ-Sandbox) provides users with the best practices for continuously producing data recipes. It features low overhead, portability, and guidance.
The sandbox is run using the following commands by default, and for more information and details, please refer to the sandbox documentation.
python tools/sandbox_starter.py --config configs/demo/sandbox/sandbox.yaml
tools/preprocess
for you to preprocess these data.
data-juicer
, you can run the commands or tools mentioned above using this docker image.# run the data processing directly
docker run --rm \ # remove container after the processing
--name dj \ # name of the container
-v <host_data_path>:<image_data_path> \ # mount data or config directory into the container
-v ~/.cache/:/root/.cache/ \ # mount the cache directory into the container to reuse caches and models (recommended)
datajuicer/data-juicer:<version_tag> \ # image to run
dj-process --config /path/to/config.yaml # similar data processing commands
# start the container
docker run -dit \ # run the container in the background
--rm \
--name dj \
-v <host_data_path>:<image_data_path> \
-v ~/.cache/:/root/.cache/ \
datajuicer/data-juicer:latest /bin/bash
# enter into this container and then you can use data-juicer in editable mode
docker exec -it <container_id> bash
Data-Juicer is released under Apache License 2.0.
We are in a rapidly developing field and greatly welcome contributions of new features, bug fixes and better documentations. Please refer to How-to Guide for Developers.
If you have any questions, please join our discussion groups.
Data-Juicer is used across various LLM products and research initiatives, including industrial LLMs from Alibaba Cloud's Tongyi, such as Dianjin for financial analysis, and Zhiwen for reading assistant, as well as the Alibaba Cloud's platform for AI (PAI). We look forward to more of your experience, suggestions and discussions for collaboration!
Data-Juicer thanks and refers to several community projects, such as Huggingface-Datasets, Bloom, RedPajama, Pile, Alpaca-Cot, Megatron-LM, DeepSpeed, Arrow, Ray, Beam, LM-Harness, HELM, ....
If you find our work useful for your research or development, please kindly cite the following paper.
@inproceedings{chen2024datajuicer,
title={Data-Juicer: A One-Stop Data Processing System for Large Language Models},
author={Daoyuan Chen and Yilun Huang and Zhijian Ma and Hesen Chen and Xuchen Pan and Ce Ge and Dawei Gao and Yuexiang Xie and Zhaoyang Liu and Jinyang Gao and Yaliang Li and Bolin Ding and Jingren Zhou},
booktitle={International Conference on Management of Data},
year={2024}
}