Arrow files too large when training SegFormer

MRNOBODY-ZST commented 1 month ago

System Info

transformers version: 4.42.4
Platform: Windows-11-10.0.22631-SP0
Python version: 3.12.4
Huggingface_hub version: 0.23.4
Safetensors version: 0.4.3
Accelerate version: not installed
Accelerate config: not found
PyTorch version (GPU?): 2.3.1+cu121 (True)
Tensorflow version (GPU?): not installed (NA)
Flax version (CPU?/GPU?/TPU?): not installed (NA)
Jax version: not installed
JaxLib version: not installed
Using GPU in script?: Fine_tune_SegFormer_on_custom_dataset.ipynb
GPU type: NVIDIA GeForce RTX 4090

Who can help?

@amyeroberts

Information

[X] The official example scripts
[X] My own modified scripts

Tasks

[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction

Make a custon dataset to train SegFormer I'm using a dataset around 33000 images, 26000 for training 7000 for validation When training started, the transformer will continuously writing cache files (with speed around 50mb/s) to C:/Users//.cache/huggingface/metrics/mean_io_u/default Which name is default_experiment-1-0.arrow When it reaches 50 epochs, the file size is about 707GB. I don't know why.

Expected behavior

Disk space drainage, the training can't be continued.

qubvel commented 1 month ago

Hi @MRNOBODY-ZST Thanks for reporting! Let me check this!

MRNOBODY-ZST commented 1 month ago

@qubvel Really Appreciate This!!! By the way, i'm using the training script in this repo.

huggingface / transformers