Megatron-LM fine-tuning: No such file or directory model_optim_rng.pt

[zxyscz]

i want use Megatron-LM fine-tuning, but i run the process, get. a error : No such file or directory model_optim_rng.pt model_optim_rng.pt

[Muennighoff]

Hmm do you have --no_load_optim and --no_load_rng in your script?

model_optim_rng.pt files are not needed and not in the checkpoint I think

[zxyscz]

Hmm do you have --no_load_optim and --no_load_rng in your script?

model_optim_rng.pt files are not needed and not in the checkpoint I think

i run now ,but i did not how to merge many chekpoint and convert to hugging face format, can you help me?

[Muennighoff]

This is the script for merging & converting: https://github.com/bigcode-project/octopack/blob/4f0e261d11d41ca62ce09204716338889b7800f4/training/convert_large.sh#L4 Let me know if it does not work for you

[zxyscz]

yes, but i should git clone which branch , is this https://github.com/bigcode-project/Megatron-LM/pull/40? , when i use mft branch , it did not run.

[Muennighoff]

Yeah that one; It is already merged into main so you can probably also use the main branch It was merged slightly after the mtf branch was created hence the code is not in the mtf branch, but you can maybe also emerge main into the mtf branch if you want to

[zxyscz]

Yeah that one; It is already merged into main so you can probably also use the main branch It was merged slightly after the mtf branch was created hence the code is not in the mtf branch, but you can maybe also emerge main into the mtf branch if you want to

thx

[zxyscz]

Yeah that one; It is already merged into main so you can probably also use the main branch It was merged slightly after the mtf branch was created hence the code is not in the mtf branch, but you can maybe also emerge main into the mtf branch if you want to

i have a question, i find that humaneval python@1 value reduced a lot after fintune.

[Muennighoff]

Yeah that one; It is already merged into main so you can probably also use the main branch It was merged slightly after the mtf branch was created hence the code is not in the mtf branch, but you can maybe also emerge main into the mtf branch if you want to

i have a question, i find that humaneval python@1 value reduced a lot after fintune.

Yeah that's why we only fine-tune for few steps, e.g. OctoCoder is only fine-tuned for 2M tokens.

[zxyscz]

Yeah that one; It is already merged into main so you can probably also use the main branch It was merged slightly after the mtf branch was created hence the code is not in the mtf branch, but you can maybe also emerge main into the mtf branch if you want to

i have a question, i find that humaneval python@1 value reduced a lot after fintune.

Yeah that's why we only fine-tune for few steps, e.g. OctoCoder is only fine-tuned for 2M tokens. In addition ,I evaluated starcoderbase-25000 step , the humaneval pass@1 value was 25%, it is lower than 30%. Is it because open source starcoderbase-megatron lm is not final checkpoint?

[Muennighoff]

What script are you using to evaluate it? That may explain the small difference. It should be the final checkpoint.

[zxyscz]

What script are you using to evaluate it? That may explain the small difference. It should be the final checkpoint.

First,I convert this checkpoint to hf format, then using greedy decoding to evaluate.

[zxyscz]

i convert to hf format , is it right ?

[Muennighoff]

Yeah that looks correct. I think for pass@1 HumanEval StarCoder is evaluated using temperature=0.2. Also I would set n_samples=20.

[zxyscz]

Yeah that looks correct. I think for pass@1 HumanEval StarCoder is evaluated using temperature=0.2. Also I would set n_samples=20. I convert megatron model to hf as shown below, loading models is slow. How can i convert model many partition like this：

[Muennighoff]

You have to shard it into multiple files when saving it

[zxyscz]

You have to shard it into multiple files when saving it

How to shard it into multiple files? Is there any code to refer to？