Valid Unicode characters are filtered out

[tom-bywild]

Describe the bug

When using text with special Unicode characters, like the “ (U+201C) Left Double Quotation, this character gets filtered out sometime after providing the input.

I report this as a bug, as such characters are valid and actually required by our clients to be usable.

I noticed two regex-replace used in the functions generateHandle and generateKebab in the string.js file, but I'm not sure they're actually the cause.

Steps to reproduce

1. Create an empty vizy document
2. Provide a text with e.g. the left double quotation: “This is a quote“
3. The input stays the same, but when querying for the content, the symbol is gone

The symbol also vanishes when reloading the draft.

Craft CMS version

Craft Pro 4.4.5

Plugin version

2.0.12

Multi-site?

No

Additional context

No response

[engram-design]

I can't seem to replicate this one, sorry. There's been similar reports in the past of things, and we do do some encoding, but valid HTML entities like this should be fine.

The editor (after saving):

And the front-end:

Can you confirm the above on your end? You'll notice that you can see the raw HTML entities when you view the source of the page (not just inspecting the elements on the page).

[tom-bywild]

Yes, it's showing up. But once I save or reload the page, all content is kept except the quote.

Please note that I also checked for invisible characters, so it's not a font issue. When using the PHP-API to read the nodes, the quote is missing as well, without any empty space as a leftover (or something similar).

---

Edit: This is reproducible on my local machine as well as on the production & staging instance. This behaviour happens only with the Vizy-block, other text inputs (like title) keep all characters.

[engram-design]

Are you able to double check the raw value saves to the content table for that entry? For example, mine shows:

[{"type":"paragraph","attrs":{"textAlign":"start"},"content":[{"type":"text","text":"“This is a quote“"}]}]

[tom-bywild]

It's not in the DB, as far as I can tell.

"html": "<p>A broken quote</p>",
"content": "[{\"type\":\"text\",\"text\":\"A broken quote\"}]",

"html": "<p>\"A valid quote\"</p>",
"content": "[{\"type\":\"text\",\"text\":\"\\\"A valid quote\\\"\"}]",

Description	Image
	Result from Graph query
	Screenshot of editor

[tom-bywild]

I now wrote a JS-hack that replaces those disappearing characters in the editor's HTML input field on any input change. This was also quite tricky, and I hope we can resolve this soon. But it was the only working mitigation I saw. This way, Vizy never receives those characters in the first place, so it's "working" from an editor's POV.

If you need access to my machine, I'm happy to schedule a call for this issue. It's one of a series of issues we had with the Vizy-plugin in the past, and it's quite severe.

[tom-bywild]

For context, https://github.com/verbb/vizy/issues/198 is related to the same project.

[engram-design]

Wondering if it’s at all possible to gain a copy of your database just in case there’s something unique about that compared go my install? Totally understand if you’re unable to provide it, but if you are you can send to support@verbb.io. I certainly want to get to the bottom of this for you.

and sorry that you’ve had so many issues as per https://github.com/verbb/vizy/issues/198 but replicating the results you’re seeing seem to be the most difficult part here!

[tom-bywild]

@engram-design Having you take a look at the DB would be 💯 Before I can send you the data, I need you to sign an NDA, which I will send to your support.

Thanks again for your service, looking forward to resolving this issue soon.

[engram-design]

No problem and thanks, I’ll get that sorted.

[engram-design]

Looks like this was similar to https://github.com/verbb/hyper/issues/48 introduced in Craft 4.4 due to the changed LitEmoji library stripping out more than just emoji's (was also stripping out some Unicode characters like special quotes and characters).

Fixed for the next release. To get this early, run composer require verbb/vizy:"dev-craft-4 as 2.0.12"

[tom-bywild]

Many thanks!

[engram-design]

Fixed in 2.1.0

[jeffaglenn]

We're still experiencing issues regarding special characters being deleted even after updating to the latest version (2.1.7).

We updated the plugin on 7/12 and the client fixed an entry that was affected before the update. Then a week later that same entry was saved again and apostrophes disappeared again. We're having trouble reproducing the issue but we can see it happening within the Revisions of the entry.

Is there anything else we can do to help narrow this down?

[engram-design]

Hmmm, I can't seem to reproduce this either. Checking the revisions of my entry, it's all looking ok to me. What sort of characters are you using?

[jeffaglenn]

The only ones noticed so far are apostrophes. I know it's nothing to go off of but here are screenshots of 2 consecutive revisions happening weeks after the plugin was updated. The entry has currently been corrected again. 🤞🏻

[engram-design]

Thanks for the clarification. So it's seemingly happening at random? As in, you can go into an entry and correct it, and it saves all okay? Will continue to look into it.