token limitation after some questions

[lijutan0]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to file this report (我已阅读并同意 Language Policy).

Dify version

0.5.2

Utility with translation issue

ollama

🌐 Language affected

China

❌ Actual phrase(s)

✔️ Expected phrase(s)

dify could answer my question from ollama.

ℹ Why is the current translation wrong

after some questions, the model of ollama could not answer questions.

[dosubot[bot]]

🤖

你好，@lijutan0！很高兴再次见到你。希望你一切都好。

Based on the provided context, it seems like the issue you're experiencing with the Ollama model in dify version 0.3.21 might be related to the tokenization process. The GPT-2 tokenizer, which is used in the Ollama model, is designed for English language and might not work correctly with Chinese language. This is because Chinese language does not have spaces between words and each character can be a word itself. Therefore, the tokenization process might not work as expected, leading to incorrect calculation of tokens.

If the number of tokens calculated for a text is greater than the context size, the text is cut off to fit the context size. This might lead to incomplete or incorrect responses, especially if the cut-off happens in the middle of a sentence or a word. The maximum number of tokens that the Ollama model in dify version 0.3.21 can handle in a single query is 4096 tokens.

However, without the complete code and more information about the _get_num_tokens_by_gpt2 method, it's hard to provide a definitive answer. If you could provide more details about the _get_num_tokens_by_gpt2 method and the exact behavior you're experiencing, it would be helpful in diagnosing the issue.

For more information, you can refer to the following sources:

• api/core/model_runtime/model_providers/ollama/text_embedding/text_embedding.py
• api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py
• api/core/model_runtime/model_providers/ollama/llm/llm.py

I hope this helps! If you have any other questions, feel free to ask.

Best, Dosu

Sources

• api/core/model_runtime/model_providers/ollama/text_embedding/text_embedding.py
• api/core/model_runtime/model_providers/__base/tokenizers/gpt2_tokenzier.py
• api/core/model_runtime/model_providers/ollama/llm/llm.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[guchenhe]

Error seems self explanatory, what context size did you input when configuring the model?

[dosubot[bot]]

Hi, @lijutan0,

I'm helping the team manage their backlog and am marking this issue as stale. From what I understand, you raised an issue about a token limitation in the Dify utility when using the "ollama" translation model for Chinese language. There were some detailed responses from both "dosubot" and "guchenhe" seeking more information about the tokenization process, the maximum number of tokens, and the context size input for the model configuration.

Could you please confirm if this issue is still relevant to the latest version of the repository? If it is, kindly let the team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation.

I (Dosu)