Add '--raise_error_at_min_scale False' flag in pretrain and finetune scripts to avoid minimum loss scale crash caused by bad batch.

TinyLLaVA / TinyLLaVA_Factory

A Framework of Small-scale Large Multimodal Models

https://arxiv.org/abs/2402.14289

Apache License 2.0

658 stars 68 forks source link

Closed RobotiX101 closed 3 months ago

RobotiX101 commented 3 months ago

To save your day, if any one encounter crash in finetune/pretrain and find this log:

exception: current loss scale already at minimum - cannot decrease scale anymore. exiting run

add ''--raise_error_at_min_scale False' flag in finetune or pretrain scripts.