Donut training slowness

NielsRogge / Transformers-Tutorials

This repository contains demos I made with the Transformers library by HuggingFace.

MIT License

8.48k stars 1.33k forks source link

Donut training slowness #317

Open khadkechetan opened 1 year ago

khadkechetan commented 1 year ago

Hi, I am currently working on fine-tuning the Donut model on custom dataset. While I train on only 2500 samples, with 30 epoch it takes around 20 hrs on 24GB GPU.

How can we speed up this process?

@NielsRogge

khadkechetan commented 10 months ago

Hi, Any response would be highly appreciated.

NielsRogge commented 10 months ago

Which batch size are you using? Can you verify the GPU is used (by typing nvidia-smi in the terminal)?

I'd recommend this guide: https://huggingface.co/docs/transformers/perf_train_gpu_one

poetaster commented 8 months ago

Hi, Any response would be highly appreciated.

I'm guessing you were training with a RTX 4090? Did you manage to get your training times down?