[P1] Support more huggingface (transformer-based) models

[frankaging]

Descriptions: Ideally, all the models listed here can be supported by this library without exposing the model details to the users of this library.

This requires we set up model folders for all model types and write config metadata for each of them annotating where to do interventions. This requires a lot of effort. This is a PR tracking the process towards the goal of supporting as many as we can.

Each model should take less than an hour to (1) configure and (2) write simple unit tests.

Here is the list of models that are in the pipeline to support (in order):

• BERT-family
  • RoBERTa
  • DeBERTa
  • ELECTRA
• xlm (multilingual model)
• t5
• Mistral
• Mixtral (MoE, MixtralForCausalLM)
• Phi
• Mamba (but need to support recurrent interventions, not just layerwise interventions)
• backpack-gpt2
• please feel free to suggest other new models to support!

[frankaging]

Update: most of the works are completed, tracked here: https://github.com/stanfordnlp/pyvene/tree/dev/models and https://github.com/stanfordnlp/pyvene/tree/peterwz

[SubramanyamSahoo]

@frankaging, I see that we have an extensive list of models to support. Would it be beneficial to prioritize them based on their popularity or relevance to the community? This could help us focus our efforts more effectively and ensure that we address the most pressing needs first.

[SubramanyamSahoo]

• models

Certainly! Here are some additional transformer-based models that could be considered for support:

GPT-3 OpenAI Codex GPT-Neo GPT-J BART (Bidirectional and Auto-Regressive Transformers) Groq