dosubot[bot]
commented
2 weeks ago To split Chinese content in a PDF using the SemanticSplitterNodeParser to make the retrieve function more accurate, you need to ensure that the sentence_splitter is capable of handling Chinese text. The default split_by_sentence_tokenizer may not be suitable for Chinese, so you should provide a custom sentence splitter that can handle Chinese sentence segmentation.
Here is an example of how you can implement a custom sentence splitter for Chinese text and use it with SemanticSplitterNodeParser:
from llama_index.core.node_parser.text.semantic_splitter import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding
import jieba
def chinese_sentence_splitter(text: str) -> List[str]:
# Use jieba to split Chinese text into sentences
sentences = jieba.lcut(text, cut_all=False)
return sentences
# Create an instance of the SemanticSplitterNodeParser with the custom sentence splitter
parser = SemanticSplitterNodeParser.from_defaults(
embed_model=OpenAIEmbedding(),
sentence_splitter=chinese_sentence_splitter
)
# Now you can use the parser to split your Chinese PDF content
# Example usage:
# document = Document(text="你的中文内容在这里")
# nodes = parser.build_semantic_nodes_from_documents([document])In this example, jieba is used to split Chinese text into sentences. You can replace jieba with any other Chinese text segmentation tool if needed. The custom chinese_sentence_splitter function is then passed to the SemanticSplitterNodeParser to handle Chinese content accurately.
Alternatively, you can use the DashScopeJsonNodeParser which is designed to handle Chinese text and provides more advanced options for splitting text:
from llama_index.node_parser.dashscope.base import DashScopeJsonNodeParser
import os
# Ensure the DashScope API key is set in the environment variables
os.environ["DASHSCOPE_API_KEY"] = "your_dashscope_api_key"
# Create an instance of the DashScopeJsonNodeParser
parser = DashScopeJsonNodeParser()
# Now you can use the parser to split your Chinese PDF content
# Example usage:
# document = Document(text="你的中文内容在这里")
# nodes = parser.build_semantic_nodes_from_documents([document])The DashScopeJsonNodeParser uses a predefined set of separators suitable for Chinese text and can handle various document formats, making it a robust choice for splitting Chinese content in PDFs [1][2][3].
To continue talking to Dosu, mention @dosu.
912100012
Question Validation
Question
"SemanticSplitterNodeParser" work in English context,but not work in Chinese.I want to know how to split the chinese content to make the "retrieve" more accurate.Please give me an idea. Thank you