CotEditor doesn't display all invisible characters

[victormoliveira1]

Description

The following characters aren't displayed as invisible characters in CotEditor, even with all checkboxes enabled in General > Invisible characters:

Next Line (Codepoint U+0085, NEL) Zero Width Joiner (Codepoint U+200D, ZWJ) Line Separator (Codepoint U+2028, LS) Paragraph Separator (Codepoint U+2029, PS) Zero-Width No-Break Space (Codepoint U+FEFF, ZWNBSP) Soft Hyphen (Codepoint U+00AD, SHY)

Operators of General Punctuation: Function Application (Codepoint U+2061, FA) Invisible Times (Codepoint U+2062, IT) Invisible Separator (Codepoint U+2063, IS) Invisible Plus (Codepoint U+2064, IP)

My observation was based from this table of non-printing characters where only NEL, ZWJ, LS, PS, ZWNBSP and SHY wasn't detected by CotEditor, and from this post / issue where only FA, IT, IS, IP (General Punctuation block) wasn't detected.

Note: In some instances, when editing the open document, it appears that CotEditor actually detects these characters. Unfortunately, this behavior was not predictable enough to be reproduced.

To Reproduce

1) Create a new document

2) Open every Unicode description mentioned above

3) Click on "Copy to the clipboard" and paste on the open document

Expected behavior

The specified characters should be marked as invisible (non-printable) characters.

CotEditor version

CotEditor 4.8.6

macOS version

MacOS Sonoma 14.5

Additional context

No response

[1024jp]

related #1462

[1024jp]

cf. The current spec in CotEditor (“Display invisible characters in CotEditor” page in the CotEditor User Guide)

[1024jp]

Thank you for the feedback.

Operators of General Punctuation: Function Application (Codepoint U+2061, FA) Invisible Times (Codepoint U+2062, IT) Invisible Separator (Codepoint U+2063, IS) Invisible Plus (Codepoint U+2064, IP)

These are currently missing from the list, but they should be drawn. I'll add them in the next update, CotEditor 4.8.7.

Regarding the rest of the codepoints, they are either already added or intentionally excluded.

Next Line (Codepoint U +0085, NEL) Line Separator (Codepoint U+2028, LS) Paragraph Separator (Codepoint U+2029, PS)

These three characters are registered and displayed as “Line ending.” You could not see those invisible characters shown because you did not correctly enter those characters. In CotEditor, all line endings are automatically replaced with the line endings in the document. This is to prevent unintentional mixing of line ending codes when the user copies and pastes text from another source while editing their document. To input a line ending other than the document's line ending:

• Perform the “Paste Exactly” command.
• Input a character by specifying its code point using the “Input In Unicode Code Point” command.

See the “Handle various types of line endings” page in the CotEditor User Guide for details.

Zero-Width No-Break Space (Codepoint U+FEFF, ZWNBSP)

This character is already on the list. It should be drawn as an invisible control character. You may have missed something. Try inputting it with the Unicode code point.

Soft Hyphen (Codepoint U+00AD, SHY)

I have previously tried but found out it is technically difficult with the current implementation because there is no way in the framework to distinguish whether the (soft) hyphen is displayed or not. See #1462 for details.

Zero Width Joiner (Codepoint U+200D, ZWJ)

I have intentionally excluded this as an invisible control character to be drawn also due to limitations in the framework.

[1024jp]

Actually, there is no specific definition of "invisible characters" to be drawn by editors, so each editor must define them on its own. I do my best, but it can be differed from other editors.