fix issue with embedding size too small

bigscience-workshop / metadata

Experiments on including metadata such as URLs, timestamps, website descriptions and HTML tags during pretraining.

Apache License 2.0

30 stars 12 forks source link

Closed jordiclive closed 1 year ago

jordiclive commented 1 year ago

Resize embeddings so they aren't too small after the entity_paragraph added special tokens.
And Moved the code block before the parameters are defined for the optimizer.