JSON Output Contains Garbled Characters for Chinese Audio Transcription

[tylike]

Environment:

• OS: Windows 11 (cmd with UTF-8 output)
• Whisper.cpp Version: 1.6.0
• Model: V3 large

Command Used:

main.exe -m ggml-large-v3.bin -of d:\VideoInfo\80\subtitle -d 60000 -osrt -ojf -otxt d:\VideoInfo\80\80.wav -l auto --prompt "这是**简体中文**内容,每一段落尽量长,使用标点符号逗号、句号、感叹号、问号、双引号等。**不要乱码**"

Issue:

• The .txt and .srt files are generated correctly.
• The .json file contains garbled/incorrect characters.

Additional Details:

• When using English audio, the .json, .srt, and .txt files are all generated correctly.

Steps to Reproduce:

1. Run the command provided with a Chinese audio file.
2. Check for garbled characters in the .json file.

en_subtitle.json en_subtitle.srt.txt 80.zip

[tamo]

~Apparently escape_double_quotes_and_backslashes is not valid for mutibyte strings.~ ~Maybe we should use replace or replace_all function for escaping.~ ~Also there may be other problems too.~

EDIT: It seems to be a tokenizer problem. Two characters (瑞典) became four tokens:

    "transcription": [
        {
...
            "text": "945年6月,瑞典著名犹太人建筑师马克思·甘佩尔接到了一份邀请函。",
            "tokens": [
...
                {
                    "text": "9",
...
                },
                {
                    "text": "45",
...
                },
                {
                    "text": "年",
...
                },
                {
                    "text": "6",
...
                },
                {
                    "text": "月",
...
                },
                {
                    "text": ",",
...
                },
                {
                    "text": "�",
...
                },
                {
                    "text": "�",
...
                },
                {
                    "text": "�",
...
                },
                {
                    "text": "�",
...
                },
                {
                    "text": "著",

Already reported here