search

Adriankhl / godot-llm

LLM in Godot
MIT License
100 stars 3 forks source link

When I use Chinese input, I get the error "Missing 2 UTF-8 continuation byte(s)". What could be the cause of this? #11

Closed hanscene closed 2 months ago

hanscene commented 2 months ago

image When I use Chinese input, I get the error "Missing 2 UTF-8 continuation byte(s)". What could be the cause of this?

Adriankhl commented 2 months ago

Ah, it is because the utf8 encoding of Chinese character might get split up in the generation process, and unlike std::string, Godot String doesn't know how to combine them correctly if they are received separately, i.e., streaming through on_generate_text_updated, can you check if the text is displayed correctly in the final on_generate_text_finished signal?

Let me figure out a solution to detect the incomplete utf8 encoding and send it out once it is valid.

Adriankhl commented 2 months ago

Now it has been fixed on the main branch 😄 Thanks for the report.

Adriankhl commented 2 months ago

I have release the new version, it is now fixed there, let me know if you have any further problem.