Phi-3 can't deal with Japanese. How can I solve this issue?

[Hideki105]

Phi-3 can't deal with Japanese

I enjoy Phi-3 which Microsoft made. I meet the error of onnxruntime_genai. How can I solve this issue ?

• Phi-3 can't deal with Japanese. How can I solve this issue?

To reproduce

I can not suggest. sorry.

Urgency

No response

Platform

Windows

OS Version

windows 10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

0.2.0rc3

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[Hideki105]

Code

import onnxruntime_genai as og
import argparse
import time

model = og.Model(".\Phi-3-mini-128k-instruct-onnx\cpu_and_mobile\cpu-int4-rtn-block-32")
tokenizer = og.Tokenizer(model)
tokenizer_stream = tokenizer.create_stream()

def input_llm(text):
    print("Question:",text)
    input_tokens = tokenizer.encode(text)
    params = og.GeneratorParams(model)
    params.try_use_cuda_graph_with_max_batch_size(1)
    params.input_ids = input_tokens
    generator = og.Generator(model, params)
    return generator

def output_llm(generator):
    print("Answer:")
    stt = time.time()
    list_error = []
    list_sentence = []
    while not generator.is_done():
        generator.compute_logits()
        generator.generate_next_token()
        new_token = generator.get_next_tokens()[0]
        if not new_token in list_error:
            try:
                list_sentence.append(tokenizer_stream.decode(new_token))
            except:
                list_error.append(new_token)
                list_sentence.append(new_token)
    print(list_sentence)
    fin = time.time()
    print(fin-stt)
    return list_error

InPut

text = "<|user|>こんにちは。データ分析するにはなにをすればいい？<|end|><|assistant|>"
generator = input_llm(text)
list_error= output_llm(generator)
print(list_error)

OutPut

Question: <|user|>こんにちは。データ分析するにはなにをすればいい？<|end|><|assistant|>
Answer:
['', 'デ', 'ー', 'タ', '分', 233, 161, 147, 'に', 'は', 'い', 'く', 'つ', 'か', 'の', 'ス', 'テ', 'ッ', 'プ', 'が', 'あ', 'り', 'ま', 'す', '。', 'ま', 'ず', '、', 'デ', 'ー', 'タ', 'セ', 'ッ', 'ト', 'を', 232, 146, 145, '集', 'し', 'ま', 'す', '。', 'こ', 'れ', 'は', '、', '必', '要', 'な', '情', '報', 'を', 174, 'む', 'デ', 'ー', 'タ', 'を', '集', 'め', 'る', 'こ', 'と', 'で', 'す', '。', '次', 'に', '、', 'デ', 'ー', 'タ', 'を', 152, 183, '理', 'し', '、', '不', '要', 'な', '部', '分', 'を', '除', '去', 'し', 'ま', 'す', '。', 'こ', 'れ', 'に', 'よ', 'り', '、', '分', 'の', 141, 188, 234, 138, 'が', '向', '上', 'し', 'ま', 'す', '。', 'そ', 'の', '後', '、', 'デ', 'ー', 'タ', 'を', '分', 'し', '、', '意', 148, 182, 'の', 'あ', 'る', '洞', 178, 162, 'や', 'パ', 'タ', 'ー', 'ン', 'を', '見', 'つ', 'け', '出', 'し', 'ま', 'す', '。', '最', '後', 'に', '、', '分', '結', '果', 'を', '解', 236, 139, 'し', '、', '意', 'を', 179, 'き', '出', 'し', '、', 'そ', 'の', '結', '果', 'を', '報', '告', 'し', 'ま', 'す', '。', 'こ', 'れ', 'ら', 'の', 'ス', 'テ', 'ッ', 'プ', 'を', 235, 187, 'む', 'こ', 'と', 'で', '、', 'デ', 'ー', 'タ', '分', 'を', '成', '長', 'さ', 'せ', 'る', 'こ', 'と', 'が', 'で', 'き', 'ま', 'す', '。', '']
63.10930633544922
[233, 161, 147, 232, 146, 145, 174, 152, 183, 141, 188, 234, 138, 148, 182, 178, 162, 236, 139, 179, 235, 187]

[yuslepukhin]

Are you feeding it UTF-8 or one of the Japanese code pages? How are you inputting your text? You are using Windows, what is your default code page on the computer? The stackoverflow issue points to the problem at the tokenization stage, before you even run the model.

[yuslepukhin]

Also for Genai issues, it would be faster to get help here: https://github.com/microsoft/onnxruntime-genai/issues

[Hideki105]

Thank you for mail.

Are you feeding it UTF-8 or one of the Japanese code pages?

I feed the Japanese Question text to Genai. text = "<|user|>こんにちは。データ分析するにはなにをすればいい？<|end|><|assistant|>".

How are you inputting your text?

I put the test code on the issue page. Please see the issue page. I make the test code. Of course, I see the below page. https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/model-qa.py

You are using Windows, what is your default code page on the computer?

My active code page is 437. I run the code on Jupyter Notebook.

Hideki

2024年4月25日(木) 3:03 Dmitri Smirnov @.***>:

Are you feeding it UTF-8 or one of the Japanese code pages? How are you inputting your text? You are using Windows, what is your default code page on the computer?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/onnxruntime/issues/20448#issuecomment-2075533824, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJ5JTTKJCB3RWOPXZB6SSNDY67XXFAVCNFSM6AAAAABGWDRSOKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZVGUZTGOBSGQ . You are receiving this because you authored the thread.Message ID: @.***>

[github-actions[bot]]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.