[Question]: unidentifiable Chinese code

912100012 commented 1 month ago

Question Validation

[X] I have searched both the documentation and discord for an answer.

Question

when i persist the property index locally,the chinese charactor is not started with the \uxxxx in utf-8 encoding,but the unidentifiable Chinese code such as “��Ҷ��ԣ��һ��ø��”，so how can i solve this problem by setting the param?@dosu

dosubot[bot] commented 1 month ago

The issue you're encountering with unidentifiable Chinese characters when persisting the property index locally is likely related to character encoding settings. Unfortunately, the current documentation does not specify any encoding parameters or settings available in the method used for persisting the property index ^[1]^[2].

However, there is a known issue related to character encoding in the current version of the codebase, specifically when creating embeddings. The encoding does not allow for certain characters, which might be causing the errors you're seeing ^[3].

To address this, you might need to ensure that the data is properly encoded in UTF-8 before persisting it. Here’s a general approach you can take:

Ensure UTF-8 Encoding: Make sure that the data you are working with is encoded in UTF-8 before you persist it. You can do this by explicitly encoding your strings in UTF-8.
Check File Handling: When writing to files, ensure that the file handling operations are set to use UTF-8 encoding. For example:
```
with open('your_file.txt', 'w', encoding='utf-8') as f:
    f.write(your_data)
```
Review Data Pipeline: Check your data pipeline to ensure that at no point the data is being converted to a different encoding that might not support Chinese characters.

If these steps do not resolve the issue, you might need to look into the specific implementation details of the persist method and see if there are any internal encoding settings that can be adjusted. If the problem persists, consider raising an issue in the repository for further assistance.

To continue talking to Dosu, mention @dosu.

912100012 commented 1 month ago

the index that i created is from the document that i manually add as follow ，rather from the file such as 'txt','pdf',so how can i specific the encoding to make the encoding 'utf-8' so that it will not appear the unidentifiable Chinese code in the output property index file.@dosu

dosubot[bot] commented 1 month ago

To specify the encoding to UTF-8 for a manually added document and prevent unidentifiable Chinese characters in the output property index file, you can include the following meta tag in the HTML document's head section:

<meta charset="utf-8" />

This tag should be placed within the <head> section of your HTML document to set the encoding to UTF-8. Here is an example:

<html>
<head>
    <meta charset="utf-8" />
    ...
</head>
<body>
    ...
</body>
</html>

Ensure that this meta tag is correctly placed within the <head> section of your HTML document to set the encoding to UTF-8 ^[1].

If you are working with Python code and manually adding documents, you can also ensure that the strings are encoded in UTF-8 before processing them. For example:

document_content = "你的文档内容".encode('utf-8').decode('utf-8')

This ensures that the document content is properly encoded in UTF-8.

To continue talking to Dosu, mention @dosu.

912100012 commented 1 month ago

the code as fllow,it will generate 5 files in storage folder:defaultvector_store.json、docstore.json、graph_store.json、imagevector_store.json、index_store.json、property_graph_store.json， why only "property_graph_store.json" file is encoding by ANSI,other file is well,it is the "utf-8" that i want。how can i solve this to make all of the file is "utf-8" encoding

index = PropertyGraphIndex.from_documents( documents, kg_extractors=[kg_extractor], llm=MySetting.my_llm_mode(), embed_model=MySetting.my_embedding_mode(), show_progress=True,

# property_graph_store=graph_store

) index.storage_context.persist("../storage/lmsg")

@dosu

dosubot[bot] commented 1 month ago

The issue you're encountering is due to the different methods used for writing property_graph_store.json compared to the other JSON files. Specifically, property_graph_store.json is written using the model_dump_json() method, which might not explicitly set the encoding to UTF-8.

To ensure all files, including property_graph_store.json, are encoded in UTF-8, you can modify the file writing operation to explicitly specify the encoding. Here’s how you can do it:

Locate the persist method in the PropertyGraphIndex class.
Modify the file writing operation to specify UTF-8 encoding.

Here’s an example of how you can modify the persist method:

def persist(
    self, persist_path: str, fs: Optional[fsspec.AbstractFileSystem] = None
) -> None:
    """Persist the graph store to a file."""
    if fs is None:
        fs = fsspec.filesystem("file")
    with fs.open(persist_path, "w", encoding="utf-8") as f:
        f.write(self.graph.model_dump_json())

By adding encoding="utf-8" to the fs.open call, you ensure that the file is written with UTF-8 encoding.

This change should make sure that property_graph_store.json is encoded in UTF-8, just like the other JSON files ^[1]^[2].

To continue talking to Dosu, mention @dosu.

912100012 commented 1 month ago

how can i specify the encoding utf8 when i use the storage_context's persist method.@dosu

run-llama / llama_index