Using GPU in script?: Fine_tune_SegFormer_on_custom_dataset.ipynb
GPU type: NVIDIA GeForce RTX 4090
Who can help?
@amyeroberts
Information
[X] The official example scripts
[X] My own modified scripts
Tasks
[X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)
Reproduction
Make a custon dataset to train SegFormer
I'm using a dataset around 33000 images, 26000 for training 7000 for validation
When training started, the transformer will continuously writing cache files (with speed around 50mb/s) to C:/Users//.cache/huggingface/metrics/mean_io_u/default
Which name is default_experiment-1-0.arrow
When it reaches 50 epochs, the file size is about 707GB.
I don't know why.
Expected behavior
Disk space drainage, the training can't be continued.
System Info
transformers
version: 4.42.4Who can help?
@amyeroberts
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Make a custon dataset to train SegFormer I'm using a dataset around 33000 images, 26000 for training 7000 for validation When training started, the transformer will continuously writing cache files (with speed around 50mb/s) to C:/Users//.cache/huggingface/metrics/mean_io_u/default
Which name is default_experiment-1-0.arrow
When it reaches 50 epochs, the file size is about 707GB.
I don't know why.
Expected behavior
Disk space drainage, the training can't be continued.