Closed guustfranssensEY closed 2 years ago
After checking the config of a supported model e.g. bert
ecco.from_pretrained('bert-base-uncased', activations=True)
lm.model_config
{'activations': ['\\d+\\.output\\.dense'],
'embedding': 'embeddings.word_embeddings',
'partial_token_prefix': '##',
'token_prefix': '',
'tokenizer_config': {'partial_token_prefix': '##', 'token_prefix': ''},
'type': 'mlm'}
I found that I had to add the following as tokenizer config
'tokenizer_config': {'partial_token_prefix': '', 'token_prefix': ''}
therefore my full config for the custom model is now:
model_config = {
'embedding': 'roberta.embeddings.word_embeddings',
'type': 'mlm',
'activations': ['intermediate\.dense'],
'token_prefix': '',
'partial_token_prefix': '',
'tokenizer_config': {'partial_token_prefix': '', 'token_prefix': ''},
}
After fixing this, my code is able to obtain the beautifull visuals @jalammar has made :)
P.S. Could the tokenizer_config be added to the documentation?
Awesome! Thanks for working through this, @guustfranssensEY. The intent was that 'tokenizer_config' is made automatically by the library (so the user doesn't repeat themselves needlessly). Nice catch finding out it's not kicking in when users supply the config object.
I think the direction next is to remove tokenizer_config
all together. I've made issue #67 to track this.
I am working on integrating my custom model vinai\bertweet-base with Ecco, however I ran into the following issue:
I created the lm in the following way:
upon further inspection I believe the error comes from the following line:
I could not traceback the origin of tokenizer_config. Therefore I assume it also has to be passed in the model_config for a custom model? If so, this needs to be specified in the docs.
Or could this issue be related in a strange way to #65