Error in text segmentation and cleaning of dify local knowledge base

[RR0810]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
• [X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing? Tell me about your story.

I tried to segment and clean the text of the uploaded local knowledge base, and selected the custom mode. There was a segment identifier, and the maximum length of the segment was set to 1000. However, the result was segmented in places that did not meet 1000, and a sentence was Separate incomplete words.

2. Additional context or comments

No response

3. Can you help us with this feature?

• [ ] I am interested in contributing to this feature.

[dosubot[bot]]

Hey @RR0810! 👋 I'm here to assist you with bugs, questions, and becoming a contributor. I'm Dosu, a bot designed to help while you wait for a human maintainer.

To resolve the issue of text segments not adhering to the specified 1000-character limit and splitting words when using the custom mode in the Dify application, you need to configure the segmentation rules properly. Specifically, you should set the "Maximum segment length" to 1000 characters and ensure that the "Segmentation identifier" is appropriately set to avoid splitting words.

Here is how you can configure it:

1. Maximum segment length: Set this to 1000 characters to ensure that no segment exceeds this length.
2. Segmentation identifier: Set an identifier such as "\n" (newline) to segment the text at appropriate points, which helps in avoiding splitting words.

Example configuration:

segmentation_rules:
  segmentation_identifier: "\n"
  maximum_segment_length: 1000
  segment_overlap_length: 100  # Optional, set overlap to retain semantic relevance

Additionally, you can use the Dify API to create documents with custom segmentation rules. Here is an example of how to create a document by text using the API:

curl --location --request POST 'https://api.dify.ai/v1/datasets/<uuid:dataset_id>/document/create_by_text' \
--header 'Authorization: Bearer {api_key}' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "Dify",
    "text": "Dify means Do it for you...",
    "indexing_technique": "high_quality",
    "process_rule": {
        "rules": {
                "pre_processing_rules": [{
                        "id": "remove_extra_spaces",
                        "enabled": true
                }, {
                        "id": "remove_urls_emails",
                        "enabled": true
                }],
                "segmentation": {
                        "separator": "###",
                        "max_tokens": 500
                }
        },
        "mode": "custom"
    }
}'

By setting these rules and using the API, the system will segment the text at specified points and ensure that each segment does not exceed 1000 characters, thus preventing the splitting of words ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[crazywoola]

@JohnJyong Please take a look at this