Epic: Test script

[hahuyhoang411]

We need to set the test script for our training pipeline.

• Data generation: @hungphongtrn

  • [ ] Check the audio generated (audio match the prompt)
  • [ ] Check the integrity of audio files with the converted dataset
  • [ ] Check the consistency.
  • [ ] Check sound tokens decode to .wav match with the original .wav file

• Training pipeline: @bachvudinh

  • [ ] Loss convergence (Loss curve png)
  • [ ] Proper weight initialization (avg distribution for new tokens)

• Inference: @bachvudinh

  • [x] Handling of special tokens (BOS, EOS, PAD) (e.g. Ensure EOS doesn't map to -100)
  • [ ] Output coherence and relevance (e.g. bleu?)
  • [ ] Check repetition problem

• (Not priority) Performance: we should only do small test (subset of the benchmark)

  • [x] MMLU-Pro
  • [x] MATH
  • [x] Human-Eval
  • [ ] ...

• (Not priority) Hyper-parameters config: This is for auto-checking the config suitable for the hardware Ref: https://huggingface.co/nanotron/old_bench/tree/main/llama-1B

  • [ ] Batchsize
  • [ ] Method: FSDP, Deepspeed
  • [ ] Learning rate
  • [ ] Warmup steps

  Updating...

[hahuyhoang411]

@bachvudinh some of the code is scattering around, you should connect with @hungphongtrn to centralize them together.