search

vgaraujov / CPC-NLP-PyTorch

Implementation of Contrastive Predictive Coding for Natural Language
10 stars 3 forks source link

Could you please share some checkpoints of the training? #3

Closed YuffieHuang closed 2 years ago

YuffieHuang commented 2 years ago

Hi @vgaraujov. I'm sorry to bother you again. The training really takes a lot of time, especially since I can only use 1 GPU now. I'm wondering if you can share any checkpoints that you trained on the BookCorpus dataset? In that case, I am able to continue training the model and fine-tune it with my own dataset. Thank you so much!

vgaraujov commented 2 years ago

Hi @YuffieHuang, I found a checkpoint, but I'm not very sure if it belongs to this figure. You can test it and let me know if it works. Here you find the link.

YuffieHuang commented 2 years ago

Hi @YuffieHuang, I found a checkpoint, but I'm not very sure if it belongs to this figure. You can test it and let me know if it works. Here you find the link.

Thank you so much! Let me check it out.

YuffieHuang commented 2 years ago

Hi @YuffieHuang, I found a checkpoint, but I'm not very sure if it belongs to this figure. You can test it and let me know if it works. Here you find the link.

@vgaraujov The sad thing happens that there seems a mismatch between the shared checkpoint and the current network structure. The shared checkpoint contains a weight named "infonce.Wk.weight". However, weights such as "Wk.0.weight", "Wk.0.bias", "Wk.1.weight", and "Wk.1.bias" are missing, which are required by the current structure of CPCv1.

vgaraujov commented 2 years ago

I just checked my local repository (fortunately, I still have it), and I found another checkpoint. I think that one is correct—the last one undoubtedly belongs to a previous version of my implementation. Please check the same link. Let me know the result.

YuffieHuang commented 2 years ago

Yes! It works! Thank you so much!