search

IBM / ibm-generative-ai

IBM-Generative-AI is a Python library built on IBM's large language model REST interface to seamlessly integrate and extend this service in Python programs.
https://ibm.github.io/ibm-generative-ai/
Apache License 2.0
250 stars 101 forks source link

LangChainInterface doesn't override get_num_tokens etc #232

Closed ind1go closed 12 months ago

ind1go commented 12 months ago

Version Information

What is the expected behavior?

Calling get_num_tokens or get_num_tokens_from_messages on a LangChainInterface would return the right number.

What is the actual behavior?

It actually uses the GPT2 tokenizer (falling back to langchain/schema/language_model.py, function get_tokenizer()), and comes out with the wrong number.

Please provide a unit test that demonstrates the bug.

llm = LangChainInterface(credentials=credentials, model="meta-llama/llama-2-70b-chat", params=params)
print(llm.get_num_tokens(prompt_text))
# ...vs...
model = Model("meta-llama/llama-2-70b-chat", params=TokenParams, credentials=credentials)
tokenized_response = model.tokenize([prompt_text], return_tokens=True)
print(tokenized_response[0].token_count)

Other notes on how to reproduce the issue?

Any possible solutions?

I note it's implemented on the chat interface, just not on plain LangChainInterface yet - so I'm registering my interest in it being fixed or fixing it there too.

I also note that get_token_ids isn't implemented for either so it would also be giving the GPT2-based result.

Can you identify the location in the GENAI source code where the problem exists?

src/genai/extensions/langchain/llm.py

If the bug is confirmed, would you be willing to submit a PR?

Yes

Tomas2D commented 12 months ago

Hey @ind1go, good catch.

I will ensure this will be delivered in the upcoming release (next week)