``CharLMEmbeddings.train()'' has been overridden thus cannot disable dropout while evaluation.

[LorrinWWW]

Due to issue #184, CharLMEmbeddings.train() has been overridden:

    def train(self, mode=True):
        pass

But this prevents it from switching to eval mode, causing that the dropout is still in effect during evaluation. Related issue #216, so calling tagger.train() won't disable dropout in CharLMEmbeddings. That explains why the evaluation score varies every time. Is it possible to fix it? Thanks a lot!

[alanakbik]

Hello @LorrinWWW yes this was a problem of the CharLMEmbeddings class, but they are deprecated since Flair 0.4 in favor of the FlairEmbeddings class and will probably be removed altogether by Flair 0.5. Could you use FlairEmbeddings instead? They should not have this error.

[LorrinWWW]

@alanakbik thanks for your reply. In fact, I encountered this problem when I was fine-tuning the ''chunking'' model since it used CharLMEmbeddings, please consider update this model as well. Furthermore, according to what I see in the source code (flair/embeddings.py line 1112), FlairEmbeddings.train() has also been overridden and thus passed, I suspect the problem still exists.

[alanakbik]

Thanks for pointing this out, we'll retrain the chunking model and put in a PR shortly.

The FlairEmbeddings should work deterministically. When they initially load the LanguageModel, it gets put into eval() disabling all dropout. The LM stays in eval() since switching the FlairEmbeddings between eval() and train() has no effect because we override the train() method.

Here is a snippet to check:

# load embeddings
embeddings = FlairEmbeddings('news-forward')
print(embeddings)

# put embeddings in train
embeddings.train()

# sentence 1
sentence = Sentence('Hello')
embeddings.embed(sentence)
print(sentence[0].embedding[:10])

# sentence 2
sentence = Sentence('Hello')
embeddings.embed(sentence)
print(sentence[0].embedding[:10])

Embeddings should always be the same, even in train mode.

[LorrinWWW]

I see, thanks a lot!