Howto do evaluation on new annotated data NER?

Question Hi there,

I trained a NER model on the conll03 dataset, and it performs quite well.

However, I want to test it on a new domain. So we annotated some new texts and brought them into conll03 format.

I can read these files with the model.inference_from_file(eval_file) just fine. It then proceeds to give me a dict with offsets, tokens and the respective label it would predict.

save_dir = "saved_models/ner_dbmdz_conll" model = Inferencer.load(save_dir) result = model.inference_from_file(debby_eval_file)

However, I would like to evaluate how well this model is doing on the new gold annotation.

I cannot figure out how to do that with Evaluator.eval. Is that even the right approach?

Your help is much appreciated.

Thanks.

Hey @tnhaider your proposed solution to use the Evaluator.eval function is exactly the right approach. We have implemented two evaluation examples for text classification and question answering in this script.

Unfortunately there is no example for evaluating NER - though it should be straightforward to adjust the existing code. Would you like give it a shot yourself? We will be able to support you in case you run into problems and if you succeed we would be more than happy to add your code to the library.

Alright, I modified the evaluate_classification to evaluate_ner.

The lang_model is my tuned ner model.

data_dir is the path to my eval file, and evaluation_filename the name of my eval file.

I changed the label list to ["[PAD]", "X", "O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC", "B-OTH", "I-OTH"]

I use a NERProcessor with a max_seq_len of 64.

For the adaptive model, I changed the task_type to 'ner' model = AdaptiveModel.convert_from_transformers(lang_model, device=device, task_type="ner")

However, it appears the data_handler cannot read my eval file.

06/09/2020 13:29:38 - INFO - transformers.tokenization_utils -   Didn't find file saved_models/ner_dbmdz_bert_conll/added_tokens.json. We won't load it.
06/09/2020 13:29:38 - INFO - transformers.tokenization_utils -   loading file saved_models/ner_dbmdz_bert_conll/vocab.txt
06/09/2020 13:29:38 - INFO - transformers.tokenization_utils -   loading file None
06/09/2020 13:29:38 - INFO - transformers.tokenization_utils -   loading file saved_models/ner_dbmdz_bert_conll/special_tokens_map.json
06/09/2020 13:29:38 - INFO - transformers.tokenization_utils -   loading file saved_models/ner_dbmdz_bert_conll/tokenizer_config.json
06/09/2020 13:29:38 - INFO - farm.data_handler.data_silo -   
Loading data into the data silo ... 
              ______
               |o  |   !
   __          |:`_|---'-.
  |__|______.-/ _ \-----.|       
 (o)(o)------'\ _ /     ( )      

Traceback (most recent call last):
  File "evaluation.py", line 190, in <module>
    evaluate_ner()
  File "evaluation.py", line 47, in evaluate_ner
    batch_size=batch_size)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/data_silo.py", line 93, in __init__
    self._load_data()
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/data_silo.py", line 191, in _load_data
    train_file = self.processor.data_dir / self.processor.train_filename
  File "/usr/lib64/python3.6/pathlib.py", line 899, in __truediv__
    return self._make_child((key,))
  File "/usr/lib64/python3.6/pathlib.py", line 686, in _make_child
    drv, root, parts = self._parse_args(args)
  File "/usr/lib64/python3.6/pathlib.py", line 640, in _parse_args
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType

AFAICT good work so far.

Please make sure that label_list and the labels you trained your NER model with are exactly the same (order matters, see here)

Regarding your issue: Could you update to current master branch of FARM, please? And also post the code you used? I suspect there is something wrong in the Processor args in combination with the Evaluator. Should be an easy fix : )

Ah, I see what the problem might be. The NERProcessor can't handle train_filename=None, while the TextClassificationProcessor was already fixed in that regard.

I can get to it in a moment.

Honestly the specific Processor does not care about any train test or dev filename being None, this is handled inside the data_silo on a global level (for all types of processors). That is why I think something is weird with how you create the NERProcessor. Could you post the code how you do it?

Apologies, I have been distracted over the last couple of days. I got the latest version of FARM, and tried to replicate the problem. However, now I get a different error.

Here is the code:

def evaluate_ner():
    ##########################
    ########## Settings
    ##########################
    device, n_gpu = initialize_device_settings(use_cuda=True)
    #lang_model = "deepset/bert-base-german-cased-sentiment-Germeval17"
    lang_model = "../../FARM_NER/examples/saved_models/ner_dbmdz_bert_conll"
    do_lower_case = False
    batch_size = 100

    data_dir = Path("../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08_1444_CONLL03/annotation/Aavik.xml/")
    evaluation_filename = "debby.corrected.conll"
    label_list = ["[PAD]", "X", "O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC", "B-OTH", "I-OTH"]
    metric = "f1_macro"

    # 1.Create a tokenizer
    tokenizer = Tokenizer.load(
        pretrained_model_name_or_path=lang_model,
        do_lower_case=do_lower_case)

    # 2. Create a DataProcessor that handles all the conversion from raw text into a pytorch Dataset
    # Here we load GermEval 2017 Data automaticaly if it is not available.

    processor = NERProcessor(
        tokenizer=tokenizer,
        max_seq_len=64,
        label_list=label_list,
        metric=metric,
        train_filename=None,
        dev_filename=None,
        dev_split=0,
        test_filename=evaluation_filename,
        data_dir=data_dir,
    )

    # 3. Create a DataSilo that loads dataset, provides DataLoaders for them and calculates a few descriptive statistics of our datasets
    data_silo = DataSilo(
        processor=processor,
        batch_size=batch_size)

    # 4. Create an Evaluator
    evaluator = Evaluator(
        data_loader=data_silo.get_data_loader("test"),
        tasks=data_silo.processor.tasks,
        device=device
    )

    # 5. Load model
    model = AdaptiveModel.convert_from_transformers(lang_model, device=device, task_type="ner")
    # use "load" if you want to use a local model that was trained with FARM
    # model = AdaptiveModel.load(lang_model, device=device)
    model.connect_heads_with_processor(data_silo.processor.tasks, require_labels=True)

    # 6. Run the Evaluator
    results = evaluator.eval(model)
    f1_score = results[0]["f1_macro"]
    print("Macro-averaged F1-Score:", f1_score)

I now get this error message:

Preprocessing Dataset ../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08
06/12/2020 18:53:24 - ERROR - farm.data_handler.processor -   Could not convert this sample to features: 

      .--.        _____                       _      
    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-5-0
Clear Text: 
    text: Von _ O O
 1925 _ O O
 bis _ O O
 1933 _ O O
 hatte _ O O
 er _ O O
 die _ O O
 musikalische _ O O
 Leitung _ O O
 am _ O O
 Estonia-Theater _ O I-ORG
 in _ O O
 Tallinn _ O I-LOC
 inne _ O O
 ; _ O O
 daneben _ O O
 dirigierte _ O O
 er _ O O
 auch _ O O
 in _ O O
 Finnland _ O B-LOC
 , _ O O
 Lettland _ O B-LOC
 und _ O O
 Ungarn _ O B-LOC
 . _ O O

    ner_label: ['Von _ O O', '1925 _ O O', 'bis _ O O', '1933 _ O O', 'hatte _ O O', 'er _ O O', 'die _ O O', 'musikalische _ O O', 'Leitung _ O O', 'am _ O O', 'Estonia-Theater _ O I-ORG', 'in _ O O', 'Tallinn _ O I-LOC', 'inne _ O O', '; _ O O', 'daneben _ O O', 'dirigierte _ O O', 'er _ O O', 'auch _ O O', 'in _ O O', 'Finnland _ O B-LOC', ', _ O O', 'Lettland _ O B-LOC', 'und _ O O', 'Ungarn _ O B-LOC', '. _ O O']
Tokenized: 
    tokens: ['Von', '_', 'O', 'O', '1925', '_', 'O', 'O', 'bis', '_', 'O', 'O', '1933', '_', 'O', 'O', 'hatte', '_', 'O', 'O', 'er', '_', 'O', 'O', 'die', '_', 'O', 'O', 'musikalische', '_', 'O', 'O', 'Leitung', '_', 'O', 'O', 'am', '_', 'O', 'O', 'Est', '##onia', '-', 'Theater', '_', 'O', 'I', '-', 'OR', '##G', 'in', '_', 'O', 'O', 'Tall', '##inn', '_', 'O', 'I', '-', 'L', '##OC']
    offsets: [0, 4, 6, 8, 11, 16, 18, 20, 23, 27, 29, 31, 34, 39, 41, 43, 46, 52, 54, 56, 59, 62, 64, 66, 69, 73, 75, 77, 80, 93, 95, 97, 100, 108, 110, 112, 115, 118, 120, 122, 125, 128, 132, 133, 141, 143, 145, 146, 147, 149, 152, 155, 157, 159, 162, 166, 170, 172, 174, 175, 176, 177]
    start_of_word: [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, False, False, False, True, True, True, False, False, False, True, True, True, True, True, False, True, True, True, False, False, False]
Features: 
    None
_____________________________________________________
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/hpc/shared/EasyBuild/apps/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/data_silo.py", line 124, in _dataset_from_chunk
    dataset = processor.dataset_from_dicts(dicts=dicts, indices=indices)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/processor.py", line 336, in dataset_from_dicts
    self._featurize_samples()
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/processor.py", line 295, in _featurize_samples
    sample.features = self._sample_to_features(sample=sample)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/processor.py", line 734, in _sample_to_features
    tokenizer=self.tokenizer,
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/input_features.py", line 167, in samples_to_features_ner
    labels_token = expand_labels(labels_word, initial_mask, non_initial_token)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/utils.py", line 410, in expand_labels
    labels_token.append(labels_word[word_index])
IndexError: list index out of range
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "evaluation.py", line 195, in <module>
    evaluate_ner()
  File "evaluation.py", line 47, in evaluate_ner
    batch_size=batch_size)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/data_silo.py", line 105, in __init__
    self._load_data()
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/data_silo.py", line 242, in _load_data
    self.data["test"], self.tensor_names = self._get_dataset(test_file)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/data_handler/data_silo.py", line 176, in _get_dataset
    for dataset, tensor_names in results:
  File "/hpc/shared/EasyBuild/apps/Python/3.7.4-GCCcore-8.3.0/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
IndexError: list index out of range

Hey no problem, this seems to be a problem related to your dataset in combination with the NERPRocessor.

The tokens (Finnland) and actual labels (B-LOC) as in "Finnland _ O B-LOC" look like whitespace separated, so you should change the parameter delimiter=" " in NERProcessor

Now the eval file is loading fine, but it can't load the NER model anymore.

I tried retraining a new model, but the error stays the same. It can't find the config.json

      .--.        _____                       _      
    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-0-0
Clear Text: 
    text: Aavik , Juhan
    ner_label: ['I-PER', 'I-PER', 'I-PER']
Tokenized: 
    tokens: ['A', '##av', '##ik', ',', 'Ju', '##han']
    offsets: [0, 1, 3, 6, 8, 10]
    start_of_word: [True, False, False, True, True, False]
Features: 
    input_ids: [3, 32, 1625, 172, 26918, 1348, 3064, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    padding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    initial_mask: [0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ner_label_ids: [1, 6, 1, 1, 6, 6, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
_____________________________________________________
06/15/2020 02:08:27 - INFO - farm.data_handler.processor -   

      .--.        _____                       _      
    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-0-0
Clear Text: 
    text: Aavik , Juhan
    ner_label: ['I-PER', 'I-PER', 'I-PER']
Tokenized: 
    tokens: ['A', '##av', '##ik', ',', 'Ju', '##han']
    offsets: [0, 1, 3, 6, 8, 10]
    start_of_word: [True, False, False, True, True, False]
Features: 
    input_ids: [3, 32, 1625, 172, 26918, 1348, 3064, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    padding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    initial_mask: [0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ner_label_ids: [1, 6, 1, 1, 6, 6, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
_____________________________________________________
Preprocessing Dataset ../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08_1444_CONLL03/annotation/Aavik.xml/debby.correc
06/15/2020 02:08:27 - INFO - farm.data_handler.data_silo -   Examples in train: 0
06/15/2020 02:08:27 - INFO - farm.data_handler.data_silo -   Examples in dev  : 0
06/15/2020 02:08:27 - INFO - farm.data_handler.data_silo -   Examples in test : 13
06/15/2020 02:08:27 - INFO - farm.data_handler.data_silo -   
06/15/2020 02:08:28 - INFO - transformers.modeling_utils -   loading weights file saved_models/bert-german-ner-base/language_model.bin from cache at saved_models/bert-german-ner-base/language_model.bin
Traceback (most recent call last):
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/configuration_utils.py", line 247, in get_config_dict
    local_files_only=local_files_only,
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/file_utils.py", line 261, in cached_path
    raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file saved_models/bert-german-ner-base/config.json not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "evaluation.py", line 196, in <module>
    evaluate_ner()
  File "evaluation.py", line 58, in evaluate_ner
    model = AdaptiveModel.convert_from_transformers(lang_model, device=device, task_type="ner")
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/adaptive_model.py", line 603, in convert_from_transformers
    ph = TokenClassificationHead.load(model_name_or_path)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/prediction_head.py", line 590, in load
    full_model = AutoModelForTokenClassification.from_pretrained(pretrained_model_name_or_path)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/modeling_auto.py", line 1115, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/configuration_auto.py", line 187, in from_pretrained
    pretrained_model_name_or_path, pretrained_config_archive_map=ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, **kwargs
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/configuration_utils.py", line 270, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load 'saved_models/bert-german-ner-base'. Make sure that:

- 'saved_models/bert-german-ner-base' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'saved_models/bert-german-ner-base' is the correct path to a directory containing a 'config.json' file

This is the content of the language model directory:

$ ls saved_models/bert-german-ner-base/
language_model.bin          prediction_head_0.bin          processor_config.json    tokenizer_config.json
language_model_config.json  prediction_head_0_config.json  special_tokens_map.json  vocab.txt

I am able to load Severin Simmlers model from Huggingface:

lang_model = "severinsimmler/literary-german-bert"

But that gives me this error:

    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-0-0
Clear Text: 
    text: Aavik , Juhan
    ner_label: ['I-PER', 'I-PER', 'I-PER']
Tokenized: 
    tokens: ['\u2581A', 'a', 'vik', '\u2581', ',', '\u2581Juha', 'n']
    offsets: [0, 1, 2, 6, 6, 8, 12]
    start_of_word: [True, False, False, True, False, True, False]
Features: 
    input_ids: [0, 62, 11, 5342, 6, 4, 90565, 19, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
    padding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    initial_mask: [0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ner_label_ids: [1, 6, 1, 1, 6, 1, 6, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
_____________________________________________________
06/15/2020 02:29:11 - INFO - farm.data_handler.processor -   

      .--.        _____                       _      
    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-0-0
Clear Text: 
    text: Aavik , Juhan
    ner_label: ['I-PER', 'I-PER', 'I-PER']
Tokenized: 
    tokens: ['\u2581A', 'a', 'vik', '\u2581', ',', '\u2581Juha', 'n']
    offsets: [0, 1, 2, 6, 6, 8, 12]
    start_of_word: [True, False, False, True, False, True, False]
Features: 
    input_ids: [0, 62, 11, 5342, 6, 4, 90565, 19, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
    padding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    initial_mask: [0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ner_label_ids: [1, 6, 1, 1, 6, 1, 6, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
_____________________________________________________
Preprocessing Dataset ../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08_1444_CONLL03/annotation/Aavik.xml/debby.correc
06/15/2020 02:29:12 - INFO - farm.data_handler.data_silo -   Examples in train: 0
06/15/2020 02:29:12 - INFO - farm.data_handler.data_silo -   Examples in dev  : 0
06/15/2020 02:29:12 - INFO - farm.data_handler.data_silo -   Examples in test : 13
06/15/2020 02:29:12 - INFO - farm.data_handler.data_silo -   
Traceback (most recent call last):
  File "evaluation.py", line 198, in <module>
    evaluate_ner()
  File "evaluation.py", line 60, in evaluate_ner
    model = AdaptiveModel.convert_from_transformers(lang_model, device=device, task_type="ner")
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/adaptive_model.py", line 586, in convert_from_transformers
    lm = LanguageModel.load(model_name_or_path)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/language_model.py", line 157, in load
    f"Model not found for {pretrained_model_name_or_path}. Either supply the local path for a saved "
Exception: Model not found for jplu/tf-xlm-r-ner-40-lang. Either supply the local path for a saved model or one of bert/roberta/xlnet/albert/distilbert models that can be downloaded from remote. Ensure that the model class name can be inferred from the directory name when loading a Transformers' model. Here's a list of available models: https://farm.deepset.ai/api/modeling.html#farm.modeling.language_model.LanguageModel.load
(new_python37-venv) (base) [thomas.haider@cn2-hpc examples]$ python evaluation.py 
06/15/2020 02:31:05 - INFO - transformers.file_utils -   PyTorch version 1.5.0+cu92 available.
06/15/2020 02:31:07 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None
06/15/2020 02:31:07 - INFO - transformers.tokenization_utils -   Model name 'severinsimmler/literary-german-bert' not found in model shortcut name list (xlm-roberta-base, xlm-roberta-large, xlm-roberta-large-finetuned-conll02-dutch, xlm-roberta-large-finetuned-conll02-spanish, xlm-roberta-large-finetuned-conll03-english, xlm-roberta-large-finetuned-conll03-german). Assuming 'severinsimmler/literary-german-bert' is a path, a model identifier, or url to a directory containing tokenizer files.
06/15/2020 02:31:08 - INFO - filelock -   Lock 139771073173712 acquired on /home/thomas.haider/.cache/torch/transformers/110292d69533565b80d9208bdddfaa7e04e6678ac95960fa24f2ff1ada35493b.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4.lock
06/15/2020 02:31:08 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/special_tokens_map.json not found in cache or force_download set to True, downloading to /home/thomas.haider/.cache/torch/transformers/tmp1rfzbn5a
Downloading: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 112/112 [00:00<00:00, 41.2kB/s]
06/15/2020 02:31:09 - INFO - transformers.file_utils -   storing https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/special_tokens_map.json in cache at /home/thomas.haider/.cache/torch/transformers/110292d69533565b80d9208bdddfaa7e04e6678ac95960fa24f2ff1ada35493b.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
06/15/2020 02:31:09 - INFO - transformers.file_utils -   creating metadata file for /home/thomas.haider/.cache/torch/transformers/110292d69533565b80d9208bdddfaa7e04e6678ac95960fa24f2ff1ada35493b.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
06/15/2020 02:31:09 - INFO - filelock -   Lock 139771073173712 released on /home/thomas.haider/.cache/torch/transformers/110292d69533565b80d9208bdddfaa7e04e6678ac95960fa24f2ff1ada35493b.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4.lock
06/15/2020 02:31:09 - INFO - filelock -   Lock 139771073173712 acquired on /home/thomas.haider/.cache/torch/transformers/3e238d4cea4c66db65e41412779102776c97d5af27078ca51bd6ff0a97181af6.3da1ead749b4edc2ee19839a764f045087f404b96939f84f7683e0c69cd27207.lock
06/15/2020 02:31:09 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/tokenizer_config.json not found in cache or force_download set to True, downloading to /home/thomas.haider/.cache/torch/transformers/tmp58novgyy
Downloading: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 152/152 [00:00<00:00, 52.4kB/s]
06/15/2020 02:31:10 - INFO - transformers.file_utils -   storing https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/tokenizer_config.json in cache at /home/thomas.haider/.cache/torch/transformers/3e238d4cea4c66db65e41412779102776c97d5af27078ca51bd6ff0a97181af6.3da1ead749b4edc2ee19839a764f045087f404b96939f84f7683e0c69cd27207
06/15/2020 02:31:10 - INFO - transformers.file_utils -   creating metadata file for /home/thomas.haider/.cache/torch/transformers/3e238d4cea4c66db65e41412779102776c97d5af27078ca51bd6ff0a97181af6.3da1ead749b4edc2ee19839a764f045087f404b96939f84f7683e0c69cd27207
06/15/2020 02:31:10 - INFO - filelock -   Lock 139771073173712 released on /home/thomas.haider/.cache/torch/transformers/3e238d4cea4c66db65e41412779102776c97d5af27078ca51bd6ff0a97181af6.3da1ead749b4edc2ee19839a764f045087f404b96939f84f7683e0c69cd27207.lock
06/15/2020 02:31:10 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/sentencepiece.bpe.model from cache at None
06/15/2020 02:31:10 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/added_tokens.json from cache at None
06/15/2020 02:31:10 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/special_tokens_map.json from cache at /home/thomas.haider/.cache/torch/transformers/110292d69533565b80d9208bdddfaa7e04e6678ac95960fa24f2ff1ada35493b.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
06/15/2020 02:31:10 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/tokenizer_config.json from cache at /home/thomas.haider/.cache/torch/transformers/3e238d4cea4c66db65e41412779102776c97d5af27078ca51bd6ff0a97181af6.3da1ead749b4edc2ee19839a764f045087f404b96939f84f7683e0c69cd27207
Traceback (most recent call last):
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 544, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/tokenization_xlm_roberta.py", line 144, in __init__
    self.sp_model.Load(str(vocab_file))
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sentencepiece.py", line 367, in Load
    return self.LoadFromFile(model_file)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sentencepiece.py", line 177, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "None": No such file or directory Error #2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "evaluation.py", line 198, in <module>
    evaluate_ner()
  File "evaluation.py", line 29, in evaluate_ner
    do_lower_case=do_lower_case)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/modeling/tokenization.py", line 94, in load
    ret = XLMRobertaTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 393, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 547, in _from_pretrained
    "Unable to load vocabulary from file. "
OSError: Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.
(new_python37-venv) (base) [thomas.haider@cn2-hpc examples]$ python evaluation.py 
06/15/2020 02:31:29 - INFO - transformers.file_utils -   PyTorch version 1.5.0+cu92 available.
06/15/2020 02:31:31 - INFO - farm.utils -   device: cuda n_gpu: 1, distributed training: False, automatic mixed precision training: None
06/15/2020 02:31:31 - INFO - farm.modeling.tokenization -   Loading tokenizer of type 'BertTokenizer'
06/15/2020 02:31:31 - INFO - transformers.tokenization_utils -   Model name 'severinsimmler/literary-german-bert' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). Assuming 'severinsimmler/literary-german-bert' is a path, a model identifier, or url to a directory containing tokenizer files.
06/15/2020 02:31:31 - INFO - filelock -   Lock 140365591673104 acquired on /home/thomas.haider/.cache/torch/transformers/27ed7ae2eedf332e55699a64bf7037e935c603f13115d91285a005cb3dd15fe1.bac90776f6fa34759f05c7387e9124c4d626300e981b5786b82f674e08f99d72.lock
06/15/2020 02:31:31 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/vocab.txt not found in cache or force_download set to True, downloading to /home/thomas.haider/.cache/torch/transformers/tmposdcrwik
Downloading: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 240k/240k [00:00<00:00, 854kB/s]
06/15/2020 02:31:32 - INFO - transformers.file_utils -   storing https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/vocab.txt in cache at /home/thomas.haider/.cache/torch/transformers/27ed7ae2eedf332e55699a64bf7037e935c603f13115d91285a005cb3dd15fe1.bac90776f6fa34759f05c7387e9124c4d626300e981b5786b82f674e08f99d72
06/15/2020 02:31:32 - INFO - transformers.file_utils -   creating metadata file for /home/thomas.haider/.cache/torch/transformers/27ed7ae2eedf332e55699a64bf7037e935c603f13115d91285a005cb3dd15fe1.bac90776f6fa34759f05c7387e9124c4d626300e981b5786b82f674e08f99d72
06/15/2020 02:31:32 - INFO - filelock -   Lock 140365591673104 released on /home/thomas.haider/.cache/torch/transformers/27ed7ae2eedf332e55699a64bf7037e935c603f13115d91285a005cb3dd15fe1.bac90776f6fa34759f05c7387e9124c4d626300e981b5786b82f674e08f99d72.lock
06/15/2020 02:31:34 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/vocab.txt from cache at /home/thomas.haider/.cache/torch/transformers/27ed7ae2eedf332e55699a64bf7037e935c603f13115d91285a005cb3dd15fe1.bac90776f6fa34759f05c7387e9124c4d626300e981b5786b82f674e08f99d72
06/15/2020 02:31:34 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/added_tokens.json from cache at None
06/15/2020 02:31:34 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/special_tokens_map.json from cache at /home/thomas.haider/.cache/torch/transformers/110292d69533565b80d9208bdddfaa7e04e6678ac95960fa24f2ff1ada35493b.275045728fbf41c11d3dae08b8742c054377e18d92cc7b72b6351152a99b64e4
06/15/2020 02:31:34 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/tokenizer_config.json from cache at /home/thomas.haider/.cache/torch/transformers/3e238d4cea4c66db65e41412779102776c97d5af27078ca51bd6ff0a97181af6.3da1ead749b4edc2ee19839a764f045087f404b96939f84f7683e0c69cd27207
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   
Loading data into the data silo ... 
              ______
               |o  |   !
   __          |:`_|---'-.
  |__|______.-/ _ \-----.|       
 (o)(o)------'\ _ /     ( )      

06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   No train set is being loaded
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   No dev set is being loaded
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   Loading test set from: ../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08_1444_CONLL03/annotation/Aavik.xml/debby.corrected.conll
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   Got ya 2 parallel workers to convert 13 dictionaries to pytorch datasets (chunksize = 5)...
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -    0    0 
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   /w\  /w\
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -   / \  / \
06/15/2020 02:31:34 - INFO - farm.data_handler.data_silo -     
Preprocessing Dataset ../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08_1444_CONLL03/annotation/Aavik.xml/debby.correc06/15/2020 02:31:35 - INFO - farm.data_handler.processor -   *** Show 2 random examples ***
06/15/2020 02:31:35 - INFO - farm.data_handler.processor -   

      .--.        _____                       _      
    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-2-0
Clear Text: 
    text: Aavik beendete das Petersburger Kons . 1907 im Fach Trompete ( Wilhelm Wurm ) und 1911 in den Fächern Theorie und Komposition ( A . Ljadov , N . Solov \u2019 ëv , J . V tols , A . Glazunov ) .
    ner_label: ['I-PER', 'O', 'O', 'I-ORG', 'I-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O', 'I-PER', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'I-PER', 'I-PER', 'I-PER', 'O', 'B-PER', 'I-PER', 'I-PER', 'I-PER', 'I-PER', 'O', 'B-PER', 'I-PER', 'I-PER', 'I-PER', 'O', 'B-PER', 'I-PER', 'I-PER', 'O', 'O']
Tokenized: 
    tokens: ['A', '##av', '##ik', 'beendete', 'das', 'Petersburg', '##er', 'Kons', '.', '1907', 'im', 'Fach', 'Tromp', '##ete', '(', 'Wilhelm', 'Wur', '##m', ')', 'und', '1911', 'in', 'den', 'Fächer', '##n', 'Theorie', 'und', 'Komposition', '(', 'A', '.', 'Lj', '##ado', '##v', ',', 'N', '.', 'Solo', '##v', '\u2019', '[UNK]', ',', 'J', '.', 'V', 'to', '##ls', ',', 'A', '.', 'Gl', '##az', '##un', '##ov', ')', '.']
    offsets: [0, 1, 3, 6, 15, 19, 29, 32, 37, 39, 44, 47, 52, 57, 61, 63, 71, 74, 76, 78, 82, 87, 90, 94, 100, 102, 110, 114, 126, 128, 130, 132, 134, 137, 139, 141, 143, 145, 149, 151, 153, 156, 158, 160, 162, 164, 166, 169, 171, 173, 175, 177, 179, 181, 184, 186]
    start_of_word: [True, False, False, True, True, True, False, True, True, True, True, True, True, False, True, True, True, False, True, True, True, True, True, True, False, True, True, True, True, True, True, True, False, False, True, True, True, True, False, True, True, True, True, True, True, True, False, True, True, True, True, False, False, False, True, True]
Features: 
    input_ids: [102, 131, 1223, 269, 8993, 199, 16624, 105, 5638, 566, 13413, 223, 2394, 21284, 11815, 201, 4202, 6309, 30895, 2530, 136, 13201, 153, 190, 23721, 30882, 10620, 136, 11565, 201, 131, 566, 26836, 7592, 30905, 818, 196, 566, 16309, 30905, 9711, 101, 818, 206, 566, 179, 1512, 235, 818, 131, 566, 1371, 1240, 111, 616, 2530, 566, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    padding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    initial_mask: [0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ner_label_ids: [1, 6, 1, 1, 2, 2, 8, 1, 8, 8, 2, 2, 2, 2, 1, 2, 6, 6, 1, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 6, 6, 6, 1, 1, 2, 5, 6, 6, 1, 6, 6, 2, 5, 6, 6, 6, 1, 2, 5, 6, 6, 1, 1, 1, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
_____________________________________________________
06/15/2020 02:31:35 - INFO - farm.data_handler.processor -   

      .--.        _____                       _      
    .'_\/_'.     / ____|                     | |     
    '. /\ .'    | (___   __ _ _ __ ___  _ __ | | ___ 
      "||"       \___ \ / _` | '_ ` _ \| '_ \| |/ _ \ 
       || /\     ____) | (_| | | | | | | |_) | |  __/
    /\ ||//\)   |_____/ \__,_|_| |_| |_| .__/|_|\___|
   (/\||/                             |_|           
______\||/___________________________________________                     

ID: train-2-0
Clear Text: 
    text: Aavik beendete das Petersburger Kons . 1907 im Fach Trompete ( Wilhelm Wurm ) und 1911 in den Fächern Theorie und Komposition ( A . Ljadov , N . Solov \u2019 ëv , J . V tols , A . Glazunov ) .
    ner_label: ['I-PER', 'O', 'O', 'I-ORG', 'I-ORG', 'I-ORG', 'O', 'O', 'O', 'O', 'O', 'I-PER', 'I-PER', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'I-PER', 'I-PER', 'I-PER', 'O', 'B-PER', 'I-PER', 'I-PER', 'I-PER', 'I-PER', 'O', 'B-PER', 'I-PER', 'I-PER', 'I-PER', 'O', 'B-PER', 'I-PER', 'I-PER', 'O', 'O']
Tokenized: 
    tokens: ['A', '##av', '##ik', 'beendete', 'das', 'Petersburg', '##er', 'Kons', '.', '1907', 'im', 'Fach', 'Tromp', '##ete', '(', 'Wilhelm', 'Wur', '##m', ')', 'und', '1911', 'in', 'den', 'Fächer', '##n', 'Theorie', 'und', 'Komposition', '(', 'A', '.', 'Lj', '##ado', '##v', ',', 'N', '.', 'Solo', '##v', '\u2019', '[UNK]', ',', 'J', '.', 'V', 'to', '##ls', ',', 'A', '.', 'Gl', '##az', '##un', '##ov', ')', '.']
    offsets: [0, 1, 3, 6, 15, 19, 29, 32, 37, 39, 44, 47, 52, 57, 61, 63, 71, 74, 76, 78, 82, 87, 90, 94, 100, 102, 110, 114, 126, 128, 130, 132, 134, 137, 139, 141, 143, 145, 149, 151, 153, 156, 158, 160, 162, 164, 166, 169, 171, 173, 175, 177, 179, 181, 184, 186]
    start_of_word: [True, False, False, True, True, True, False, True, True, True, True, True, True, False, True, True, True, False, True, True, True, True, True, True, False, True, True, True, True, True, True, True, False, False, True, True, True, True, False, True, True, True, True, True, True, True, False, True, True, True, True, False, False, False, True, True]
Features: 
    input_ids: [102, 131, 1223, 269, 8993, 199, 16624, 105, 5638, 566, 13413, 223, 2394, 21284, 11815, 201, 4202, 6309, 30895, 2530, 136, 13201, 153, 190, 23721, 30882, 10620, 136, 11565, 201, 131, 566, 26836, 7592, 30905, 818, 196, 566, 16309, 30905, 9711, 101, 818, 206, 566, 179, 1512, 235, 818, 131, 566, 1371, 1240, 111, 616, 2530, 566, 103, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    padding_mask: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    segment_ids: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    initial_mask: [0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
    ner_label_ids: [1, 6, 1, 1, 2, 2, 8, 1, 8, 8, 2, 2, 2, 2, 1, 2, 6, 6, 1, 2, 2, 2, 2, 2, 2, 1, 2, 2, 2, 2, 6, 6, 6, 1, 1, 2, 5, 6, 6, 1, 6, 6, 2, 5, 6, 6, 6, 1, 2, 5, 6, 6, 1, 1, 1, 2, 2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
_____________________________________________________
Preprocessing Dataset ../../FARM_NER/data/mgg/MGG+Entities_project_2020-06-08_1444_CONLL03/annotation/Aavik.xml/debby.correc
06/15/2020 02:31:35 - INFO - farm.data_handler.data_silo -   Examples in train: 0
06/15/2020 02:31:35 - INFO - farm.data_handler.data_silo -   Examples in dev  : 0
06/15/2020 02:31:35 - INFO - farm.data_handler.data_silo -   Examples in test : 13
06/15/2020 02:31:35 - INFO - farm.data_handler.data_silo -   
06/15/2020 02:31:36 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/pytorch_model.bin from cache at /home/thomas.haider/.cache/torch/transformers/5c582a40aa5227576deb9d7e87d78c6dfc922d86cc083579115cc26ab3a6cc0f.e3523914d087fc098baf5025a8185cdc893c23e88dce4a165fc90e1c7413ec05
06/15/2020 02:31:38 - INFO - farm.modeling.language_model -   Automatically detected language from language model name: german
06/15/2020 02:31:39 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/severinsimmler/literary-german-bert/pytorch_model.bin from cache at /home/thomas.haider/.cache/torch/transformers/5c582a40aa5227576deb9d7e87d78c6dfc922d86cc083579115cc26ab3a6cc0f.e3523914d087fc098baf5025a8185cdc893c23e88dce4a165fc90e1c7413ec05
06/15/2020 02:31:42 - WARNING - farm.modeling.prediction_head -   `layer_dims` will be deprecated in future releases
06/15/2020 02:31:42 - INFO - farm.modeling.prediction_head -   Prediction head initialized with size [768, 3]
Evaluating: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1/1 [00:00<00:00,  1.14it/s]
Traceback (most recent call last):
  File "evaluation.py", line 198, in <module>
    evaluate_ner()
  File "evaluation.py", line 66, in evaluate_ner
    results = evaluator.eval(model)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/eval.py", line 100, in eval
    compute_metrics(metric=head.metric, preds=preds_all[head_num], labels=label_all[head_num]
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/evaluation/metrics.py", line 88, in compute_metrics
    return f1_macro(preds, labels)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/evaluation/metrics.py", line 62, in f1_macro
    return {"f1_macro": f1_score(y_true=labels, y_pred=preds, average="macro")}
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1047, in f1_score
    zero_division=zero_division)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1175, in fbeta_score
    zero_division=zero_division)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1434, in precision_recall_fscore_support
    pos_label)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1250, in _check_set_wise_labels
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 82, in _check_targets
    type_true = type_of_target(y_true)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/multiclass.py", line 263, in type_of_target
    raise ValueError('You appear to be using a legacy multi-label data'
ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.

regarding not finding the config.json It seems you are using the wrong code for loading the model. You are using model = AdaptiveModel.convert_from_transformers(lang_model, device=device, task_type="ner") for loading a model trained with FARM. We are aware that this interoperability of FARM and transformers can be difficult - but we also documented it in most places. Please read through the comments and documentation carefully before posting issues next time (this is from the code you posted and are using for loading the saved model):


    # 5. Load model
    model = AdaptiveModel.convert_from_transformers(lang_model, device=device, task_type="ner")
    # use "load" if you want to use a local model that was trained with FARM
    # model = AdaptiveModel.load(lang_model, device=device)

Well, which code should I be using?

This ist just the code that was already in evaluate.py

It also has this Mulitlabel Binarizer Problem when I load the model with the other code:

Traceback (most recent call last):
  File "evaluation.py", line 199, in <module>
    evaluate_ner()
  File "evaluation.py", line 67, in evaluate_ner
    results = evaluator.eval(model)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/eval.py", line 100, in eval
    compute_metrics(metric=head.metric, preds=preds_all[head_num], labels=label_all[head_num]
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/evaluation/metrics.py", line 88, in compute_metrics
    return f1_macro(preds, labels)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/poetry/farm/FARM/farm/evaluation/metrics.py", line 62, in f1_macro
    return {"f1_macro": f1_score(y_true=labels, y_pred=preds, average="macro")}
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1047, in f1_score
    zero_division=zero_division)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1175, in fbeta_score
    zero_division=zero_division)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/validation.py", line 73, in inner_f
    return f(**kwargs)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1434, in precision_recall_fscore_support
    pos_label)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 1250, in _check_set_wise_labels
    y_type, y_true, y_pred = _check_targets(y_true, y_pred)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/metrics/_classification.py", line 82, in _check_targets
    type_true = type_of_target(y_true)
  File "/mnt/beegfs/users/thomas.haider/Documents/workspace/new_python37-venv/lib/python3.7/site-packages/sklearn/utils/multiclass.py", line 263, in type_of_target
    raise ValueError('You appear to be using a legacy multi-label data'
ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.

I figured it out. I need to use metric="seq_f1"

Nice, seq f1 is the metric you need for NER - though I am sure there are also other problems with this model in FARM. Regarding your earlier question: AdaptiveModel.convert_from_transformers() converts a model from the NLP library transformers Because you have trained your model with FARM you should use the code:


#use "load" if you want to use a local model that was trained with FARM
model = AdaptiveModel.load(lang_model, device=device)

Seems fixed, closing now

deepset-ai / FARM

Howto do evaluation on new annotated data NER? #404