Quickfix: Accelerate YAML and LoRA Fused Ops - Githubissues

foundation-model-stack / fms-acceleration

🚀 Collection of libraries used with fms-hf-tuning to accelerate fine-tuning and training of large models.

Apache License 2.0

6 stars 12 forks source link

Quickfix: Accelerate YAML and LoRA Fused Ops #92

Closed fabianlim closed 1 month ago

fabianlim commented 1 month ago

This PR fixes:

make accelerate.yaml compatible with v1
fixed error introduced during #86 when generalizing LoRA fusedOps for Granite (that has biases).
relaxed the datatypes, allow Full-FT-FOAK and PEFT-FOAK to run in native (i.e. not mixed) precision.
update full-ft FOAK benches that were not updated in #79. Updated mixed precision settings in bench to match the state of benches before #79.

Restoration of Benches for Full FT and Regular Peft

We have removed mixed precision for these benches and restored performance, see outliers.csv

Updated Compatibility Matrix

Compatibility Matrix with Mixed Precision torch_dtype	Mixed Precision	Full-FT-FOAK	PEFT-FOAK	QPEFT-FOAK
FLOAT16	-	Compatible	Compatible	✗
FLOAT16	FP16	ValueError: Attempting to unscale FP16 gradients. See here	Compatible	Compatible
BFLOAT16	-	Compatible	Compatible	✗
BFLOAT16	BF16	Compatible	Compatible	Less Performant