question: tutorial for adding a transformer to library

[nikky4D]

HI

would you have a tutorial for how to add a transformer to the huggingface library? I would like to work with mdetr but I don't know how to get started working with such language+image models with huggingface

[NielsRogge]

Hi,

This is definitely on my list! Generally, I do the following:

• first, it's important to understand the original model. For this, I typically scan the paper, read the blog post (if there is one), check the YouTube video (if there is one).
• next, I fork the original Github repository, and create an inference script that allows to forward a single text/image/image-text pair/... through the model to get the logits/final hidden states
• next, I use the CookieCutter template to automatically create a bunch of files for me that are required for a new model to be added to HuggingFace Transformers.
• next, I start to implement the model, i.e. I implement the modeling_xxx.py file.
• next, I create a conversion script, to load the weights of the original model into my HuggingFace implementation. An example of this can be found here.
• next, I debug, to make sure the model returns the same logits/hidden states as the original one on the same input text/image/image-text pair.... I do this bottom-up, i.e. I start with the lowest layer of the network (typically the embedding layer), and then go up the network. Debugging is done using print statements (yes! It's super easy to print out intermediate hidden states and see if they are the same)
• next, I create a corresponding tokenizer/feature extractor/processor for the model, i.e. I implement tokenization_xxx.py or feature_extraction_xxx.py
• next, I implement the tests (i.e. test_modeling_xxx.py and test_tokenization_xxx.py in case the model has a tokenizer).

The following links are very helpful:

• https://huggingface.co/docs/transformers/contributing
• https://huggingface.co/docs/transformers/add_new_model

[AbdulDD]

@NielsRogge Hi there, First, Thanks for really nice and easy tutorials on multiple transformer models. I have used your training file from transformers tutorials and trained DONUT on CORD. now I have printed encoder last hidden state but now want to print encoder layer 2 or 1. How can I do that?