Support for custom embedding

adapter-hub / adapters

A Unified Library for Parameter-Efficient and Modular Transfer Learning

https://docs.adapterhub.ml

Apache License 2.0

2.54k stars 339 forks source link

Support for custom embedding #238

Closed hchoi-moveworks closed 2 years ago

hchoi-moveworks commented 2 years ago

Does AdapterHub support model with custom embeddings? That is, let us imagine a model where

all attention layers are same as Huggingface BERT model that Adapterhub supports
BertEmbedding1 is replaced with custom embedding module.

With this as base model, would adapterhub work? If adding adapters only impact attention blocks of the base model, I believe this should work, but I wanted to confirm as there maybe internal details that I do not know.

JoPfeiff commented 2 years ago

This definitely works and we are currently working on adding this type of modularity (similar to the different prediction heads). This is explicitly what we did in our paper UNKs everywhere, to be presented at EMNLP 2021. This should be available (at least in a branch) in the upcoming weeks.