feat: decoder-only models

lean-dojo / ReProver

Retrieval-Augmented Theorem Provers for Lean

https://leandojo.org

MIT License

218 stars 48 forks source link

Closed ZIYU-DEEP closed 3 months ago

ZIYU-DEEP commented 3 months ago

Adding support for decoder-only models.

[x] Update model, datamodule and other utils
[x] Update training configs
[ ] Improve the gpuprover (the current implementation can lead to oom during validation)
[ ] Check compatibility with retrievers

yangky11 commented 3 months ago

Hi Ziyu,

Thanks for the PR. We decided to move the training of decoder-only models to external tools such as LLaMA-Factory