Error with Mistral

[SalvatoreRa]

Hi,

very nice work, I have tried this notebook:

https://github.com/labmlai/inspectus/blob/main/notebooks/hf_phi.ipynb

While this is working well, I have tried to replicate just changing model:

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.1",
    trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.1",
    trust_remote_code=True)

and I encounter this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[17], line 1
----> 1 attention(res['attentions'], tokens)

TypeError: 'module' object is not callable

[lakshith-403]

The issue is not with the model change. When you run the package from the source you need to install it.

git clone git@github.com:labmlai/inspectus.git
cd python
pip install -e .

You can also follow the instructions here: https://github.com/labmlai/inspectus/blob/main/development.md

[lakshith-403]

Also if you're just trying out the package It's better to use the pip package. And frequently update it.

pip install inspectus -U

[SalvatoreRa]

I have actually installed from pip

I used the same code in the tutorial, just changed the model (for the other model it works great)

[lakshith-403]

In that case the issue might have been the existence of an internal module by the same name. We have updated the pip package and the notebooks. Can you try again after updating the package and restarting the notebook with the changes?

import inspectus

inspectus.attention(attn, tokens)

https://github.com/labmlai/inspectus/commit/b7cb00ea5e5b71e1007bcc2c93e23508794d40a7

[SalvatoreRa]

After I have deleted and reinstalled the package it is working. Thank you!

I see it is an active development, will be implemented new features?

By the way, I am preparing a review about interpretability, I would like to cite your package. How I should cite it?

[lakshith-403]

Glad to hear that! Yes we're planning to continuously add new features.

Please cite the library using the following BibTeX entry.

@misc{inspectus,
 author = {Varuna Jayasiri, Lakshith Nishshanke},
 title = {inspectus: A visualization and analytics tool for large language models},
 year = {2024},
 url = {https://github.com/labmlai/inspectus},
}