SFT improvements (labeling fixes, different packing implementations)

EleutherAI / gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

https://www.eleuther.ai/

Apache License 2.0

6.96k stars 1.02k forks source link

Closed dmahan93 closed 3 months ago

dmahan93 commented 5 months ago

add different packing impl (Unpacked, packing until overflow), a bit naive but for SFT it shouldn't be an issue
fix labels to also have valid/test implementations
fix label masking in _get_batch to also include anything from get_ltor_masks_and_position_ids