search

mosaicml / llm-foundry

LLM training code for Databricks foundation models
https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm
Apache License 2.0
3.84k stars 503 forks source link

Why is there a warmup in hf_generate.py? #1271

Closed palash04 closed 3 weeks ago

palash04 commented 3 weeks ago

I see there is a warmup in hf_generate.py - 

if args.warmup and (not done_warmup):
    print('Warming up...')
    _ = _generate(encoded_inp)
    done_warmup = True

What is the purpose of this?

dakinggg commented 3 weeks ago

If you are doing performance benchmarking, you generally don't want to measure the first calls, but do some warmup and then measure the steady state.