search

mistralai / mistral-inference

Official inference library for Mistral models
https://mistral.ai/
Apache License 2.0
9.49k stars 835 forks source link

Code complete? #57

Open zhoumengbo opened 10 months ago

zhoumengbo commented 10 months ago

I'd like to know if the code in this repository is complete. Has anyone tried pre-training this model from scratch?

mdingemanse commented 10 months ago

for pre-training from scratch you'd need the pretraining data to be specified and available, which it isn't; therefore the code is by definition not complete.

zhoumengbo commented 10 months ago

for pre-training from scratch you'd need the pretraining data to be specified and available, which it isn't; therefore the code is by definition not complete.

Yes, you are right. Because I want to train Mistral 7B from scratch using a different dataset, I'm eager to know the integrity of the model code. If you are familiar with the code and can inform me about it, I would be immensely grateful.

mdingemanse commented 10 months ago

I'm not connected to Mistral and have the same questions as you. From what I can see the repo is set up to share some code for running released versions of Mistral (and if you trawl the issues you see that it is not complete for even that, eg people cannot get the instruct version to run).

It does not look like this repo contains the code you would need to pretrain from scratch.