ebanalyse / NERDA

Framework for fine-tuning pretrained transformers for Named-Entity Recognition (NER) tasks
MIT License
154 stars 35 forks source link

tag_scheme should be able to contain the outside_tag #38

Open rubmz opened 2 years ago

rubmz commented 2 years ago

If someone passes the outside_tag as part of the tag scheme they would probably get something like:

[Expected input batch_size (324) to match target batch_size (4)]

That is because the scheme_tag would inflate the batch size by some extra bytes (the direct portion of the outside tag from the number of other schema tags). This can easily be fixed by putting in the training.py [138:0]:

    tag_complete = list(set([tag_outside] + tag_scheme))