bigcode-project / Megatron-LM

Ongoing research training transformer models at scale
Other
376 stars 49 forks source link

Train an Encoder on the BigCode Dataset #5

Closed cakiki closed 1 month ago

cakiki commented 2 years ago

Opening this after we discussed it in Slack.

It might prove useful for evaluation to train an encoder model on the same dataset used to train the main model.

Things proposed: