How to use pertained embedding as feature in the user model

[zhifeng-huang]

Hello,

I have a signature embedding of size 256 for each user. How can I use this vector as a feature in the user model?

Any tips are appreciated.

[caesarjuly]

You can use these embeddings as the initial weights for your user embedding layer. An example as

        embedding_layer = layers.Embedding(
            input_dim=xxx,
            output_dim=xxx,
            weights=[your embedding],
            trainable=True,
        )

[ddofer]

Is this compatible with the examples in the tutorials that use string Lookup? Can this be c ombined with pretrained embedding extractors (like BERT) dynamically, or only if all are extracted in advance? Can we combine that with the tutorial examples/e.g. retrieval task? Thanks!

[caesarjuly]

@ddofer This is a different approach vs. the string lookup in the tutorial.

1. For this approach, basically you are extracted the embeddings beforehand, you can use any kind of pre-train embedding including Bert.
2. For the string lookup approach, it's generating one unique embedding for each user or movie ID. It's trained dynamically with the model
3. If you want to embed Bert models to the retrieval, you can leverage other libraries and import the Bert layers to your model structure, then trained it dynamically with your retrieval

[ddofer]

Great thanks! (I've added that to my retrieval model using strong, large (384 dim) pretrained embeddings. Oddly, results are the same (despite this being a sparse problem for users and items)

[caesarjuly]

This question cannot be answer until you share more details. The reason can be related to multiple factors.

1. What kind of pretrained embeddings are you using?
2. How sparse it is?
3. How many data you have and how complex is the model?
4. Are the pretrained embeddings fine-tuned dynamically with the model?
5. What kind of features you already have?