Combining flair embedings with build_vocab (torchtext)

[antgr]

Hi, we can use glove embedding when building vocab, using something like:

MIN_FREQ = 2

TEXT.build_vocab(train_data, 
                 min_freq = MIN_FREQ,
                 vectors = "glove.6B.300d",
                 unk_init = torch.Tensor.normal_)

We also can create embeddings using flair library, using for example:

embedding_types: List[TokenEmbeddings] = [

    WordEmbeddings('glove'),
    FlairEmbeddings('news-forward'),
    FlairEmbeddings('news-backward'),
    ELMoEmbeddings(),
    BertEmbeddings('bert-base-uncased'),
]

embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)

Could I use the above embeddings from flair instead of glove in the above code?

[antgr]

@alanakbik I have a difficulty on combining flair with regular pytorch. I think that there is a barrier between the two, and I wonder how I could use them together. Is this discouraged?

[Edresson]

@antgr Have you been successful in combining flair with regular pytorh? I'm trying to do the same!

[antgr]

Hi @Edresson , I hope you are well! Here is the best and only approach I have for this: https://github.com/pytorch/text/issues/650

[Edresson]

@antgr Thanks for the answer :).

Did you encounter any problems with contextual embeddings on the flair?

I believe that this approach will work well for the glove but I'm not sure if it will work with contextual embeddings (the same word will have 2 different embeddings depending on the context)

[antgr]

@Edresson I didn't even try it. I used transformers instead. Maybe I will come back to this in the future. But I believe that it will work, because it will substitute the word with one of the choices based on the context as you said.

[Edresson]

@tabergma @alanakbik @pommedeterresautee @stefan-it @yahshibu is there any way to combine flair with torchtext/nn.Embeddings?

We need a simple example of integration between flair and pytorch !!   I believe that would be welcome. Like me and @antgr many others would like to integrate Flair into their pytorch application. But at the moment it seems difficult.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.