Question regarding Electra Word Emmbeddings

[ChrisDelClea]

Hi guys,

i just discoverd this amazing library! My question is, i have a fine-tuned electra model and want to get the word emmbeddings out as you showed in: https://github.com/JohnSnowLabs/nlu/blob/master/examples/colab/component_examples/sentence_embeddings/NLU_ELECTRA_sentence_embeddings_and_t-SNE_visualization_example.ipynb

Is there a way i could plug-in my own model?

Best regards Chris

[maziyarpanahi]

Hi @ChrisChross

Out of curiosity, where did you get the original/pre-trained ELECTRA model and how did you fine-tune it? (which library/framework)

[ChrisDelClea]

Hi @maziyarpanahi ,

it's from Hugginface 🤗 and it's for German. You can find it here: https://huggingface.co/dbmdz/electra-base-german-europeana-cased-generator. I have fine-tuned it with simple transformers and want to get the word embeddings out of it now. Any ideas how to do it? I actually thought it might be possible with NLU.

Best regards Chris

[C-K-Loan]

HI @ChrisChross thank you for sharing. This is right now not yet possible to do straight forward with NLU. Our Electra models are originally from TF-Hub and our classes are wrapped around a TF implementation of Electra, inside of the TensorflowBert.scala class. The basic strategy would be the following.

1. Export model to TF format
2. Ensure Tensors exist that follow the naming Schema that TensorflowBert.scala follows, i.e

   TokenIdsKey = "input_ids:0"
   MaskIdsKey = "input_mask:0"
   SegmentIdsKey = "segment_ids:0"
   EmbeddingsKey = "sequence_output:0"
   SentenceEmbeddingsKey = "pooled_output:0"

3. Export TF weights and load via Spark-NLP in scala and then export it as a Spark-NLP model. The final Spark-NLP is then also loadable via NLU.

It is likely, that NLU might extend its component abstractions to also capture HF models and Pytorch models in the future. But right now, this is the most straightforward path that exists.

[maziyarpanahi]

@ChrisChross As Christian mentioned, this is currently not possible for TensorFlow v2 models. They have to be converted to TF v1 first. However, in 2-3 weeks we will release Spark NLP that is both compatible with TF v2 models automatically and it is easy to import any models (raw or fine-tune) from HuggingFace to Spark NLP via saved_model feature recently added to TF models in HF.

[ChrisDelClea]

thx