Problem starting AutoConfig from pretrained config

grantdelozier commented 2 years ago

Hello, thank you for the fantastic paper, It was a joy to read. I am attempting to instantiate LiLTRobertaLikeForTokenClassification() in my own google colab notebook to try out on my own data, but seeing an error when attempting to build the config object.

[/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    398         config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
    399         if "model_type" in config_dict:
--> 400             config_class = CONFIG_MAPPING[config_dict["model_type"]]
    401             return config_class.from_dict(config_dict, **kwargs)
    402         else:

KeyError: 'liltrobertalike'

To recreate :

from transformers import AutoConfig

pretrained_model_path = 'path to unzipped pretrain folder'

config = AutoConfig.from_pretrained(
    pretrained_model_path,
    num_labels=max(experiment_config['encode_labels'].values())+1,
    finetuning_task='funsd',
    cache_dir='/content/drive/Shareddrives/Machine Learning/pretrained_models',
    revision='v1-test',
    use_auth_token=None,
)

Which produces this error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-73-77ce32bcce05>](https://localhost:8080/#) in <module>()
      5     cache_dir='/content/drive/Shareddrives/Machine Learning/pretrained_models',
      6     revision='v1-test',
----> 7     use_auth_token=None,
      8 )

[/usr/local/lib/python3.7/dist-packages/transformers/models/auto/configuration_auto.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    398         config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
    399         if "model_type" in config_dict:
--> 400             config_class = CONFIG_MAPPING[config_dict["model_type"]]
    401             return config_class.from_dict(config_dict, **kwargs)
    402         else:

KeyError: 'liltrobertalike'

Any suggestions on how to instantiate the config for your pretrained model? It seems AutoConfig does not like this line in the config:

"model_type": "liltrobertalike"

grantdelozier commented 2 years ago

It seems the relevant bit of code for this to work is in LiLTfinetune/__init__.py

I was able to instantiate LiLTRobertaLikeForTokenClassification() from it's pretrained state in my own notebook with the following code:

from collections import OrderedDict
import os

from transformers import CONFIG_MAPPING, MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING, MODEL_NAMES_MAPPING, TOKENIZER_MAPPING, AutoConfig
from transformers.convert_slow_tokenizer import SLOW_TO_FAST_CONVERTERS, BertConverter, RobertaConverter, XLMRobertaConverter
from transformers.models.auto.modeling_auto import auto_class_factory

from LiLTfinetune.models.LiLTRobertaLike import (
    LiLTRobertaLikeConfig,
    LiLTRobertaLikeForRelationExtraction,
    LiLTRobertaLikeForTokenClassification,
    LiLTRobertaLikeTokenizer,
    LiLTRobertaLikeTokenizerFast,
)

CONFIG_MAPPING.update([("liltrobertalike", LiLTRobertaLikeConfig),])
MODEL_NAMES_MAPPING.update([("liltrobertalike", "LiLTRobertaLike"),])
TOKENIZER_MAPPING.update(
    [
        (LiLTRobertaLikeConfig, (LiLTRobertaLikeTokenizer, LiLTRobertaLikeTokenizerFast)),
    ]
)

SLOW_TO_FAST_CONVERTERS.update({"LiLTRobertaLikeTokenizer": RobertaConverter,})

MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING.update(
    [(LiLTRobertaLikeConfig, LiLTRobertaLikeForTokenClassification),]
)

AutoModelForTokenClassification = auto_class_factory(
    "AutoModelForTokenClassification", MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING, head_doc="token classification"
)

model = AutoModelForTokenClassification.from_pretrained(
    pretrained_model_path,
    from_tf=False,
    cache_dir='/content/drive/Shareddrives/Machine Learning/pretrained_models',
    revision='main',
    use_auth_token=None,
)

jpWang commented 2 years ago

Hi, glad to see you solve it so quickly.

jpWang / LiLT

Problem starting AutoConfig from pretrained config #5