token prefix in roberta model?

jalammar / ecco

Explain, analyze, and visualize NLP language models. Ecco creates interactive visualizations directly in Jupyter notebooks explaining the behavior of Transformer-based language models (like GPT2, BERT, RoBERTA, T5, and T0).

https://ecco.readthedocs.io

BSD 3-Clause "New" or "Revised" License

1.96k stars 167 forks source link

token prefix in roberta model? #73

Closed sarthusarth closed 1 year ago

sarthusarth commented 2 years ago

Trying to use a custom trained Roberta model by loading the config file but getting the error the token prefix is not present in the config. Any idea how to fix it?

guustfranssensEY commented 2 years ago

Is the token prefix specified in your config.json file?

jalammar commented 2 years ago

Yeah as @guustfranssensEY mentioned, make sure you have the prefix. This is an example from model-config.yaml:

roberta-base:
    embedding: 'embeddings.word_embeddings'
    type: 'mlm'
    activations:
        - '\d+\.output\.dense'
    token_prefix: 'Ġ'
    partial_token_prefix: ''

sarthusarth commented 2 years ago

Thanks now I can load my model. But I get this error now, what am I missing ? FireShot Capture 001 - analysis-Copy1 - Jupyter Notebook - localhost

jalammar commented 2 years ago

That's an odd and unfamiliar error. Were you able to resolve it?