search

DCDmllm / Cheetah

BSD 3-Clause "New" or "Revised" License
356 stars 35 forks source link

test_cheetah_vicuna producing non-sense output #17

Closed BennoKrojer closed 9 months ago

BennoKrojer commented 9 months ago

Hello!

I am excited to use this work in my research but am struggling to set it up.

Similar to a previous issue (https://github.com/DCDmllm/Cheetah/issues/15), I followed the instructions for downloading checkpoints and changing the config, and was able to start the test_cheetah_vicuna script. However the output is random characters. The only ambiguous part of the instructions is this:

Besides, Cheetor reuses the pretrained Q-former from BLIP-2 that matches FlanT5-XXL.

Am I supposed to also download this and change a config? That is very unclear. When I run the script it says all the weights are loaded so I am not sure what could be missing or what is happening here...

BennoKrojer commented 9 months ago

After changing the vicuna version to what you are linking to in the repository, I now get a different issue like this one: https://github.com/DCDmllm/Cheetah/issues/13

Traceback (most recent call last): File "test_cheetah_vicuna.py", line 57, in llm_message = chat.answer(raw_img_list, context) File "/home/nlp/users/bkroje/edit-from-video/change_descriptions/Cheetah/Cheetah/cheetah/conversation/conversation.py", line 253, in answer output_text = self.model.llama_tokenizer.decode(output_token, add_special_tokens=False) File "/home/nlp/users/bkroje/.conda/envs/cheetah/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3525, in decode return self._decode( File "/home/nlp/users/bkroje/.conda/envs/cheetah/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 931, in _decode filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens) File "/home/nlp/users/bkroje/.conda/envs/cheetah/lib/python3.8/site-packages/transformers/tokenization_utils.py", line 912, in convert_ids_to_tokens tokens.append(self._convert_id_to_token(index)) File "/home/nlp/users/bkroje/.conda/envs/cheetah/lib/python3.8/site-packages/transformers/models/llama/tokenization_llama.py", line 204, in _convert_id_to_token token = self.sp_model.IdToPiece(index) File "/home/nlp/users/bkroje/.conda/envs/cheetah/lib/python3.8/site-packages/sentencepiece/init.py", line 1045, in _batched_func return _func(self, arg) File "/home/nlp/users/bkroje/.conda/envs/cheetah/lib/python3.8/site-packages/sentencepiece/init.py", line 1038, in _func raise IndexError('piece id is out of range.') IndexError: piece id is out of range.

BennoKrojer commented 9 months ago

After switching from Vicuna to Llama-2, I am back to random tokens being produced...