`flash-attn` fix + new Frameworks on Sunspot

[saforem2]

• Changes:

  • Set default "export LR_WARMUP_FRAC="${LR_WARMUP_FRAC:-0.05}" which will warmup the learning rate over the first 5% of the total training iterations
  • Add ability to specify LR_DECAY_ITERS during training.
    • will default to None if not specified, according to the default from megatron/arguments.py
  • Polaris:
    • move AWS NCCL OFI plugin script to ALCF/aws_ofi_nccl_plugin.sh

  • Sunspot:

    • add support for new frameworks release: anl_24_q2_release
      • Update ALCF/sunspot-env.sh to reflect this change

    • adds fix for flash-attn discrepancy:

      Loss Curves: