When I use a Japanese PDF as a knowledge, It garbled.

[mihit]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Prepare a Japanese PDF contains "の". Use it as a knowledge.

✔️ Expected Behavior

Never garbled.

❌ Actual Behavior

Bah, it garbled as "䛾". There appears to be more than this.

[dosubot[bot]]

To prevent text from becoming garbled when using a Japanese PDF as a knowledge base in Dify version 0.6.11, ensure that the text extraction process supports Japanese characters. Here are some steps to help:

1. PDF Encoding: Ensure the PDF is properly encoded with Japanese text.
2. Text Extraction Tool: Verify that the text extraction tool or library used by Dify supports Japanese characters.

Dify supports text extraction from various document formats, including PDFs. If issues persist, you might need to check the encoding settings or use a different text extraction tool that better supports Japanese text ^[1].

Additionally, you can adjust configuration settings within Dify to improve text extraction accuracy for Japanese PDFs. Specifically, you can use custom segmentation and preprocessing rules:

• Segmentation Rules:

  • Set a segmentation identifier such as "\n".
  • Define the maximum segment length.
  • Set the number of overlapping characters between segments.

• Preprocessing Rules:

  • Replace continuous spaces, newlines, and tabs.
  • Delete all URLs and email addresses ^[2].

For enhanced preprocessing capabilities, consider using optional ETL solutions like Dify ETL or Unstructured ETL ^[2].

To continue talking to Dosu, mention @dosu.

[JohnJyong]

hi @mihit ,after testing, some Japanese characters may be recognized incorrectly. If it is convenient, can you send us your document for testing? thanks in advace~

[mihit]

太宰治 走れメロス.pdf @JohnJyong Thank you for reply. My file is private one so I can’t share it. But I send an other Japanese PDF of literary works in the public domain for testing. Hope this helps!