xlsx can recognize the same amount of data exported from the knowledge base, but csv cannot

[JayHsu5]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.11

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

1. Create a knowledge base
2. Csv file import
3. Copy csv data to xlsx

✔️ Expected Behavior

Both csv and xlsx files can count characters

❌ Actual Behavior

csv cannot

[dosubot[bot]]

The issue where CSV files cannot count characters correctly while XLSX files can in Dify version 0.6.11 is due to differences in the ETL (Extract, Transform, Load) solutions used for parsing these file formats. Different ETL solutions may have variations in file extraction effects, leading to inconsistencies in character counting between CSV and XLSX files ^[1].

To address this, you can try the following steps:

1. Preprocessing CSV Files: Ensure that the CSV file is appropriately segmented and cleaned before importing. This includes:

   • Replacing continuous spaces, newlines, and tabs.
   • Deleting all URLs and email addresses.

2. Customizing Segmentation Rules: Adjust the segmentation rules in Dify ETL to better handle CSV files. This includes setting segmentation identifiers, maximum segment length, and segment overlap length to maintain semantic relevance.

3. Encoding Handling: Ensure that the encoding of the CSV file is correctly detected. Both Dify ETL and Unstructured ETL use autodetect_encoding=True for CSV files, which attempts to automatically detect the encoding. Verify that the encoding is correctly detected and handled.

For more detailed information on Unstructured ETL’s data processing methods, refer to the official documentation ^[1].

If the issue persists, consider checking for any updates or fixes in the latest version of Dify or reviewing the error logs for more insights.

To continue talking to Dosu, mention @dosu.

[JohnJyong]

pls attach some screenshot about this issue @JayHsu5

[JayHsu5]

[JohnJyong]

If it is convenient, you can upload the desensitized version of your csv file. because form your screenshot and desc ， I can't figure out what the problem is ， thanks @JayHsu5