search

ZHZisZZ / weak-to-strong-search

[NeurIPS'24] Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models
https://arxiv.org/abs/2405.19262
36 stars 3 forks source link

Error: Not implemented error occurred about "_reorder_cache" #5

Closed andyclsr closed 2 hours ago

andyclsr commented 2 hours ago

Hi, thanks for your excellent work and open-sourcing code. I used the model of llama2-7b-hf to conduct decoding but met an error about the function _reorder_cache. The version of transformers I used is 4.46.1.

The details are reported below: https://github.com/ZHZisZZ/weak-to-strong-search/blob/060e457a9ba79d23d6e8fa695c713e1ada5cd64e/src/inference_time_alignment/decoders/cbs.py#L37 NotImplementedError: Make sure that a _reorder_cache function is correctly implemented in transformers.models.llama.modeling_llama to enable beam search for <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>

I wonder if it is the problem of llama2-7b-hf or if I can disable the use of _reorder_cache by removing this line below: https://github.com/ZHZisZZ/weak-to-strong-search/blob/060e457a9ba79d23d6e8fa695c713e1ada5cd64e/src/inference_time_alignment/decoders/cbs.py#L194

Hope for your reply and sincere thanks in advance!

ZHZisZZ commented 2 hours ago

Please use transformers==4.41.2 as required in requirements.txt. Our implementation depends on specific internal features of this version (e.g., _reorder_cache) and may not work with other versions.

The _reorder_cache is important for the algorithm to work so removing it will probably lead to unexpected results.

andyclsr commented 1 hour ago

Thanks for your detailed explanations! It really helps!