search

spcl / graph-of-thoughts

Official Implementation of "Graph of Thoughts: Solving Elaborate Problems with Large Language Models"
https://arxiv.org/pdf/2308.09687.pdf
Other
2.19k stars 159 forks source link

llama7b-hf sorts numbers #2

Closed lihuiliullh closed 1 year ago

lihuiliullh commented 1 year ago

I tried to use llama7b-hf to sort numbers according to the code in readme by changing chatGPT to llama7b-hf

lm = controller.Llama2HF( "graph_of_thoughts/controller/config.json", model_name="llama7b-hf" )

But I got the following results

WARNING:root:Could not parse step answer: Sure, I'd be happy to help you sort the list of numbers! Can you please provide the list of numbers for me to work with?. Returning empty list.

May I know whether this is a correct outcome?

nblach commented 1 year ago

Unfortunately, this is the expected outcome because Llama2-7b is not able to follow instructions as well as, for instance, GPT3.5 or GPT4. We were able to reproduce the issue and ensured that the prompt to the model is correct, but as mentioned, due to lacking capabilities, the LLM incorrectly asks for the input instead of producing a sorted list. For that reason the parser is unable to retrieve a list from the response and returns an empty list instead. On a side note, if you try to use the 70b model, it should work more smoothly.