stream_chat with tools raises a json.decoder.JSONDecodeError

[codingl2k1]

System Info / 系統信息

Transformers 4.41.2 Use the latest THUDM/glm-4-9b-chat from huggingface

Who can help? / 谁可以帮助到您？

The generated content may not be a valid json:

Information / 问题信息

• [ ] The official example scripts / 官方的示例脚本
• [ ] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

Passing tools to the stream_chat API, if the generated arg string is not a valid json, it may raises the json.decoder.JSONDecodeError

  File "/home/admin/.cache/huggingface/modules/transformers_modules/glm4-chat-pytorch-9b/modeling_chatglm.py", line 1023, in stream_chat
    response, new_history = self.process_response(response, history)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/.cache/huggingface/modules/transformers_modules/glm4-chat-pytorch-9b/modeling_chatglm.py", line 957, in process_response
    parameters = json.loads(content)
                 ^^^^^^^^^^^^^^^^^^^
  File "/home/admin/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

Expected behavior / 期待表现

If the generated output can't be parsed as a json, return a plain text instead.

[zRzRzRzRzRzRzR]

If the returned value is not json, then the tool call must have failed. It is impossible to return normal content. Therefore, under normal logic, this place must be a json, otherwise it will not be able to execute later.

[codingl2k1]

If the returned value is not json, then the tool call must have failed. It is impossible to return normal content. Therefore, under normal logic, this place must be a json, otherwise it will not be able to execute later.

The model may generate a prompt for the user. e.g. there is a tool matching the request, however, it requires more information to call it.

[codingl2k1]

Is it possible that the output stream is not completed. Then it decodes json failed.

[zRzRzRzRzRzRzR]

You are right, this code has been deprecated. In all GLM-4 demos, you only need to use the generate function provided by transformers. These codes are left over from the early GLM-3 Demo. We will delete some of the codes. Sorry.

[codingl2k1]

You are right, this code has been deprecated. In all GLM-4 demos, you only need to use the generate function provided by transformers. These codes are left over from the early GLM-3 Demo. We will delete some of the codes. Sorry.

Got it. Thank you.