Layer dimension mismatch

[BoyuanJackChen]

I'm trying to reproduce the results with Llama-2-7b-chat-hf, but I had the following error. I wonder if there was a mismatch in the expected model. Below is my command line:

python -u reproduce.py\
    --question_path="./datasets/advBench.csv"\
    --MODEL_NAME="Llama-2-7b-chat"\
    --MODEL_PATH="$HOME/huggingface_cache/models--meta-llama--Llama-2-7b-chat-hf/snapshots/_"\
    --REPHRASE_PATH="$HOME/huggingface_cache/models--meta-llama--Llama-2-7b-chat-hf/snapshots/_"\
    --SORTING_PATH="sorting.pth"\
    --EMBED_PATH="$HOME/huggingface_cache/models--sentence-transformers--gtr-t5-xl/snapshots/_"\
    --JUDGE_PATH="$HOME/huggingface_cache/models--meta-llama--Meta-Llama-Guard-2-8B/snapshots/_"\
    --question_path="./datasets/advBench.csv"\
    --n_devices=2\
    --N=2000\
    --m=5\
    --n=1

And this is the error I received:

Traceback (most recent call last):
  File "$HOME/JailMine/reproduce.py", line 50, in <module>
    miner.run(questions = questions_set,
  File "$HOME/JailMine/JailMine/core.py", line 501, in run
    i_list = self.LogitsManipulation(question=question, target=targets[i], len_of_prefix=m, num_of_prefix=N)
  File "$HOME/JailMine/JailMine/core.py", line 260, in LogitsManipulation
    val_outputs = self.sorting(torch.tensor(embedding).to(self.device))
  File "$HOME/conda-envs/jailmine/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "$HOME/conda-envs/jailmine/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "$HOME/JailMine/JailMine/core.py", line 45, in forward
    x = self.layer1(x)
  File "$HOME/conda-envs/jailmine/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "$HOME/conda-envs/jailmine/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "$HOME/conda-envs/jailmine/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1024 and 768x512)

[yuxili19]

@BoyuanJackChen Could you please provide more information about your environment and your server? I haven't encountered this question. You can run the following code to check if your gtr-x5-xl model structure is consistent with the structure in the figure.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('$HOME/huggingface_cache/models--sentence-transformers--gtr-t5-xl/snapshots/_')
print(model)

In my opinion, the Dense layer may disappear in your model. You can redownload this model with the following code and save it locally:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('sentence-transformers/gtr-t5-xl')
model.save('<your_local_path>')

After that, replace '--EMBED_PATH' with this path will be fine.

Besides, my conda environment is shown below:

[BoyuanJackChen]

@yuxili19 Thanks for the reply above! I got the code running! Below are the changes I made:

1. Previously I set SentenceTransformer's dir to the related snapshot folder in huggingface cache, and the contents in taht folder appeared to be insufficient. Hence, I used the recommended line model.save('<your_local_path>'), and it worked like a charm!
2. In core.py, input_ids is on cpu rather than gpu. Directly running the current version would lead to error. Hence, I added .to(model.device) after input_ids = tokenizer.encode(sen1+sen2+sen3+sen4+sen5+sen6+sen7, return_tensors='pt')
3. This is just a question. My current environment is the same as requirements.txt, where transformers==4.39.2, transformer-lens==2.1.0. With the above changes, I got the code running. Should we switch to your env as you shared?