UTF-8 handling in shell (hashtag2744)

[vikramsubramanian]

If i type some utf-8(non-ascii characters in the shell) followed by some white spaces, a weird period . will appear after the text. E.g. I type: Firstly: type: 这是一串中文字符 Then type: multiple white spaces.

The period appears.

kuzu> 这是一串中文字符。   

The period is generated automatically. )

[mayil-ai[bot]]

Summary: Shell adds a weird period after typing UTF-8 characters followed by white spaces.

Possible Solution

Based on the provided information, the issue seems to be related to UTF-8 character handling in the shell, particularly when non-ASCII characters are followed by white spaces, leading to an automatic period generation. To address this issue:

• Review and possibly update the process_shell_input function in tools/shell/embedded_shell.cpp to ensure it correctly handles UTF-8 characters and white spaces without introducing unintended characters like periods.
• Check the utf8proc library usage in tools/shell/embedded_shell.cpp and utf8proc_wrapper.cpp to ensure that UTF-8 characters are being processed correctly.
• Investigate if the automatic period generation is a result of incorrect handling of UTF-8 characters in utf8proc or if it's an issue with the shell's input processing logic.
• Verify that the utf8proc library is correctly included and used in tools/shell/embedded_shell.cpp and that the library itself does not contain bugs related to UTF-8 processing.
• Ensure that the shell's locale settings are configured to support UTF-8 encoding.
• Examine the display_output function for any logic that might incorrectly append periods after processing non-ASCII characters.
• Look for any regex or string manipulation in tools/shell/embedded_shell.cpp that might be causing this issue, especially after UTF-8 characters and white spaces.
• Check for any configuration settings or known issues in the shell's issue tracker that might be related to character encoding and the automatic period generation problem.
• If the issue persists, consider adding debug statements or using a debugger to trace the exact point where the period is being added to the output.

Code snippets to check

• tools/shell/embedded_shell.cpp

  The issue occurs in the shell, and this file is part of the shell implementation where input handling and display might be managed.

• third_party/utf8proc/utf8proc_wrapper.cpp

  The shell includes 'utf8proc_wrapper.h', which suggests that UTF-8 processing is done through this wrapper, and the issue is related to UTF-8 character handling.

• third_party/utf8proc/include/utf8proc.h

  This header file is likely included by other parts of the code that deal with UTF-8 processing, which is relevant to the issue.

• third_party/utf8proc/utf8proc.cpp

  This file likely contains the implementation of UTF-8 processing functions, which could be where the bug originates.

• third_party/nlohmann_json/json.hpp

  The case for UTF8_ACCEPT might be relevant if the JSON library is used in the context of the shell for handling UTF-8 input.