Closed Bakser closed 3 weeks ago
@Bakser would you provide an unit test for one of the model type you changed? e.g., llama
thanks. i can add in a new unit test based on your script.
Sure, I can help with that.
I'm not quite familiar with the unit tests of this repo. Do you mean something like the tests/integration_tests/InterventionWithGPT2TestCase.py
but for llama?
Sure, I can help with that.
I'm not quite familiar with the unit tests of this repo. Do you mean something like the
tests/integration_tests/InterventionWithGPT2TestCase.py
but for llama?
yes! that would be great! and you can initialize a much smaller llama for test! e.g., just a single layer llama for instance, since we want the unit test to be quick. thanks
@Bakser hey! any updates on the progress? thanks!
I didn't work on it on the weekend but I think I can finish it today. sorry for worrying
I find that I underestimated the workload. I will try to finish it in a couple of days.
@frankaging I've finished the test for Llama. Sorry for the delay since I was on travel.
Basically, I just copied the tests in InterventionWithGPT2TestCase.py
into InterventionWithLlamaTestCase.py
and added the implementations for Llama forward process into tests/utils.py
as like for GPT2 (but I do think if we want to implement tests for more models we need to split this file).
It can be run with python -m unittest tests.integration_tests.InterventionWithLlamaTestCase
and the output should be like:
'pyvene' is not installed.
PASS: pyvene is not installed. Testing local dev code.
=== Test Suite: InterventionWithLlamaTestCase ===
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
loaded model
testing stream: head_attention_value_output with multiple heads positions
testing stream: head_query_output with multiple heads positions
testing stream: head_key_output with multiple heads positions
testing stream: head_value_output with multiple heads positions
.
----------------------------------------------------------------------
Ran 1 test in 37.738s
OK
@frankaging I've finished the test for Llama. Sorry for the delay since I was on travel.
Basically, I just copied the tests in
InterventionWithGPT2TestCase.py
intoInterventionWithLlamaTestCase.py
and added the implementations for Llama forward process intotests/utils.py
as like for GPT2 (but I do think if we want to implement tests for more models we need to split this file).It can be run with
python -m unittest tests.integration_tests.InterventionWithLlamaTestCase
and the output should be like:'pyvene' is not installed. PASS: pyvene is not installed. Testing local dev code. === Test Suite: InterventionWithLlamaTestCase === You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 loaded model testing stream: head_attention_value_output with multiple heads positions testing stream: head_query_output with multiple heads positions testing stream: head_key_output with multiple heads positions testing stream: head_value_output with multiple heads positions . ---------------------------------------------------------------------- Ran 1 test in 37.738s OK
Thanks! @Bakser
Description
Fix the bug reported in #158 by modifying the modeling scripts of
gemma
,gpt_neo
,gpt_neox
,llama
,llava
,mistral
, mainly involving three points:split_head_and_permute
operation to split the hidden representations by attention heads_lm_type_to_module_mapping
), which missed elements inv[1:]
Testing Done
All tests passed.
Checklist:
[Your Priority] Your Title