search

DeNederlandscheBank / nqm

A Transformer-based Machine for answering questions on insurance companies
MIT License
0 stars 0 forks source link

Large file storage #13

Closed jm-glowienke closed 3 years ago

jm-glowienke commented 3 years ago

The checkpoint files of a transformer model are quite large. The ones from the IWSLT_en_de architecture of fairseq are around 400MB, but these can grow bigger up to 2GB or more. The problem is that Github only allows files of size up to 100MB in a repo.

Accordingly, a solution has to be found how to achieve the checkpoint files and share them. This is relevant to allow others to use the model and continue training it on own data.

https://git-lfs.github.com/ could be a possible solution, but have to check what is desired by DNB as well.

jm-glowienke commented 3 years ago

Process started by requesting DNB sharefile

jm-glowienke commented 3 years ago

Request correctly submitted via the rigth channel

jm-glowienke commented 3 years ago

Relevant model files uploaded!