UTF-8 handling in shell

[meido]

If i type some utf-8(non-ascii characters in the shell) followed by some white spaces, a weird period . will appear after the text. E.g. I type: Firstly: type: 这是一串中文字符 Then type: multiple white spaces.

The period appears.

kuzu> 这是一串中文字符。
The period is generated automatically.

[mayil-ai[bot]]

Summary: Shell adds a period after entering non-ASCII characters followed by white spaces.

Possible Solution

Based on the provided information, the issue seems to be related to the handling of UTF-8 input and the automatic insertion of a period character when multiple white spaces are entered after non-ASCII characters. To address this issue:

• Review and modify the process_shell_input function in embedded_shell.cpp to ensure that it correctly handles UTF-8 encoded non-ASCII characters and does not automatically insert a period after white spaces.
• Check the logic within process_shell_input or any other related input handling functions for any code that might be responsible for inserting a period after a sequence of white spaces. Look for regex patterns, string manipulation, or conditional statements that could be causing this behavior.
• Ensure that the utf8proc library is being used correctly to process UTF-8 characters. This may involve checking the usage of functions from utf8proc.h and utf8proc_wrapper.h within embedded_shell.cpp.
• Verify that the shell prompt configuration (PS1), locale settings, and terminal emulator settings are not contributing to the issue. This might involve checking environment variables and configuration files outside of the provided code snippets.
• If the issue persists, consider adding debug statements or logging within the input processing and rendering functions to trace the exact point where the period is being inserted.

Code snippets to check

• tools/shell/embedded_shell.cpp

  This file includes the shell implementation where the issue with UTF-8 handling and whitespace occurs.

• third_party/utf8proc/utf8proc_wrapper.cpp

  This file may contain the wrapper functions for UTF-8 processing which could be relevant to the issue.

• third_party/utf8proc/include/utf8proc_data.h

  This file includes data used by utf8proc which might be relevant to how UTF-8 characters are processed.

• third_party/nlohmann_json/json.hpp

  This file handles encoding of special characters in JSON and could potentially affect how whitespace and UTF-8 characters are processed.