Open stefan-it opened 5 months ago
Hi, the conversion script in LLM Foundry is not intended for MosaicBERT, which still lives here in examples repo. To export it properly with the code files, you'll need to do some manual movement of the code files. See my other answer as well: https://github.com/mosaicml/examples/issues/401#issuecomment-1629846290
Hi,
we could sucessfully pretrain various MosaicBERT models and evaluations with composer-based fine-tuning look really good :)
However, when using a/the conversion script
llm-foundry/scripts/inference/convert_composer_to_hf.py
the converted HF model seems to be initialized randomly and the MLM predictions are looking super random.I used the conversion script from the
llm-foundry
repository like this:It then shows, that various weights are not correctly initalized:
Is there any special conversion script/hints for converting a MosaicBERT composer checkpoint :thinking:
Any help is highly appreciated!