Implementation of trainer class (fp16 training & gradient accumulation)

urchade / GLiNER

Generalist and Lightweight Model for Named Entity Recognition (Extract any entity types from texts) @ NAACL 2024

https://arxiv.org/abs/2311.08526

Apache License 2.0

1.48k stars 127 forks source link

Closed MarcusLoppe closed 6 months ago

MarcusLoppe commented 6 months ago

Hi,

I've created a trainer class that has support for:

Accelerate which enables mixed precision training (fp16, bf16 or fp8).
Gradient Accumulation, which enables you to train faster and with a bigger effective batch size.
Train using total of epochs or steps
Checkpoints every X epoch/steps
Evaluation every X steps

I used the trainer in https://github.com/lucidrains/meshgpt-pytorch as a boilerplate, so credits to @lucidrains for the nicely structured trainer.