Open RefluxNing opened 1 year ago
I think you can pre-train such a model. I have tried this in my preliminary experiments. You will get reasonable outputs, but it indeed takes quite long to train til converge.
Actually I tried, but I got "Cannot allocate memory" error when using your code during preprocessing. Would you please share your model checkpoints pretrained on wiki?
sorry, I dont have a good checkpoint, this was a preliminary experiment that I did in the early stage of the project, and the model architecture was also a bit different. But I will look into your problem when I got a chance. Could you provide detailed error msg?
I saw this error when I tried to train a model on my own text file (100MB).
I see some "simple_wiki" in the code. Did you intent to train a diffusion LM on wiki? Is it too difficult (require more computing resources, for example) to pre-train such a model?