search

google-deepmind / language_modeling_is_compression

Apache License 2.0
101 stars 14 forks source link

is it possible to release your params.npz used in the paper? #7

Closed zhu-zaifei closed 11 months ago

zhu-zaifei commented 11 months ago

Hi Anianruoss,

I'm doing the local training to reproduce your result with a tiny lm. it took hours to do the training in a AWS EC2 host with GPU which cost is ~10$. the cost to train a Chinchilla 7B will be crazy expensive .....

Is it possible to release your params.npz used in the paper? If yes, we can skip the training phase and jump to the reproducing much quickly. just need some more compression to see if any more training should go.

thanks,

anianruoss commented 11 months ago

Yes, training a Chinchilla-like model is very expensive, which is why we cannot release the weights, unfortunately. However, we just added results with LLaMA1, which is open-source, to the paper and they are comparable to the Chinchilla results.