[lit-html] HTML entities in a part are shown as they are without decoding

Principles

- No HTML entities in JSON

- XML entities (subset of HTML entities) in XLIFF

Notes

Fix for demo/gulpfile.js to decode HTML entities in JSON might have unexpected side effects on Safari browsers, which have some Unicode literal string issues (whose details unknown).

Root Cause

It seems this is the expected behavior of lit-html.
lit-html sets textNode.data of HTML by parts, not innerHTML.
lit-html sets innerHTML by non-parts literals, while all the UI strings are converted to parts by I18N-ready html``
demo/gulpfile.js JavaScript preprocessor does not decode HTML entities in JSON

Reproducible Code

html`HTML entities follow ${'&lt;&gt;&#128521;'}`; // shown as &lt;&gt;&#128521;

Exception: In <i18n-format> template string, HTML entities are decoded as <i18n-format> converts them.
Preprocessed HTML template literals include HTML entities in JSON which will not be decoded in UI rendering.

Considerations on potential fixes

To allow HTML entities in JSON at runtime, all string parts must be preprocessed by html([str])
- Expected overheads
- How to detect string parts? In which phase?
- Extra preprocessing on all the string parts on each rendering
Conversion of JSON at the build time is more realistic if XLIFF is not used
- Convert HTML entities to corresponding UTF-8 strings in JSON
Conversion for ", &, ', <, &gt, and codepoint entities is done by xliff-conv if XLIFF is used
-   may need automatic conversion to and from   for visual recognition in XLIFF
- ¢, £, ¥, €, ©, ® are not supported by xliff-conv. Raw Unicode characters or their codepoint HTML entities have to be used.
- XML(XLIFF) supports only these named entities: ", &, ', <, &gt
  https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references#Predefined_entities_in_XML

Workaround for translated strings

Use XML entities (subset of HTML entities) in XLIFF, then they are encoded in UTF-8 for JSON

  <trans-unit id="lit-clock.div_3">
    <source>HTML entity &lt;>😉</source>
    <target state="needs-translation">HTML entity &lt;&gt;&#128521;</target>
  </trans-unit>

{
"div_3": "HTML entity <>😉"
}

Decode HTML entities by he --decode or equivalent tools

Workaround for parameters

Encode the part with html without binding

html`HTML entities follow ${html`&lt;&gt;&#128521;`}`; // shown as <>😉

const $en = (str) => html([str]); // Encoding function
let param = '&lt;&gt;&#128521;'; // Parameter with HTML entities
html`HTML entities follow ${$en(param)}`; // shown as <>😉

More examples

A raw non-breaking space is not visually distinguishable from a space but the code point is preserved
  is not supported in XLIFF.   can be used instead.
Codepoint entities like 😉, &#160 are recognized in XLIFF but automatically converted to their corresponding UTF-8 strings by xliff-conv

Name	Source	Preprocess	JSON	XLIFF	HTML	UI
non-breaking space
non-breaking space
non-breaking space
non-breaking space	N/A	N/A
non-breaking space (broken)	N/A	N/A		N/A
less than	<	<	<	<	<	<
less than	<	<	<	<	<	<
less than	<	<	<	<	<	<
less than (broken)	N/A	N/A	<	N/A	&lt;	<
greater than	>	>	>	>	>	>
greater than	>	>	>	>	>	>
greater than	>	>	>	>	>	>
greater than (broken)	N/A	N/A	>	N/A	&gt;	>
ampersand	&	&	&	&	&	&
ampersand	&	&	&	&	&	&
ampersand	&	&	&	&	&	&
ampersand (broken)	N/A	N/A	&	N/A	&	&
double quotation mark	"	"	\"	"	"	"
double quotation mark	"	"	\"	"	"	"
double quotation mark	"	"	\"	"	"	"
double quotation mark	N/A	N/A	\"	"	"	"
double quotation mark (broken)	N/A	N/A	"	N/A	&quot;	"
single quotation mark	'	\'	'	'	'	'
single quotation mark	'	\'	'	'	'	'
single quotation mark	'	\'	'	'	'	'
single quotation mark	N/A	N/A	'	'	'	'
single quotation mark (broken)	N/A	N/A	'	N/A	&apos;	'
cent	¢	\xA2	¢	¢	¢	¢
cent	¢	\xA2	¢	¢	¢	¢
cent	¢	\xA2	¢	¢	¢	¢
cent (broken)	N/A	N/A	¢	N/A	&cent;	¢
pound	£	\xA3	£	£	£	£
pound	£	\xA3	£	£	£	£
pound	£	\xA3	£	£	£	£
pound (broken)	N/A	N/A	£	N/A	&pound;	£
yen	¥	\xA5	¥	¥	¥	¥
yen	¥	\xA5	¥	¥	¥	¥
yen	¥	\xA5	¥	¥	¥	¥
yen (broken)	N/A	N/A	¥	N/A	&yen;	¥
euro	€	\u20AC	€	€	€	€
euro	€	\u20AC	€	€	€	€
euro	€	\u20AC	€	€	€	€
euro (broken)	N/A	N/A	€	N/A	&euro;	€
euro (broken)	N/A	N/A	\\u20AC	\u20AC	\u20AC	\u20AC
copyright	©	\xA9	©	©	©	©
copyright	©	\xA9	©	©	©	©
copyright	©	\xA9	©	©	©	©
copyright (broken)	N/A	N/A	©	N/A	&copy;	©
registered trademark	®	\xAE	®	®	®	®
registered trademark	®	\xAE	®	®	®	®
registered trademark	®	\xAE	®	®	®	®
registered trademark (broken)	N/A	N/A	®	N/A	&reg;	®
winking face emoji	😉	\uD83D \uDE09	😉	😉	😉	😉
winking face emoji	😉	\uD83D \uDE09	😉	😉	😉	😉
winking face emoji	N/A	N/A	\uD83D \uDE09	N/A	😉	😉
winking face emoji	N/A	N/A	😉	😉	😉	😉
winking face emoji (broken)	N/A	N/A	😉	N/A	😉	😉

Raw Source Code

    return html`${bind(this)}
      <div>message with HTML entities &lt;&#128521;&gt;&quot;&amp; <br>
  non-breaking space  &nbsp;  &#160; <br>
< less than &lt;  &#60; <br>
> greater than  &gt;  &#62; <br>
& ampersand &amp; &#38; <br>
" double quotation mark &quot;  &#34; <br>
' single quotation mark (apostrophe)  &apos;  &#39; <br>
¢ cent  &cent;  &#162; <br>
£ pound &pound; &#163; <br>
¥ yen &yen; &#165; <br>
€ euro  &euro;  &#8364; <br>
© copyright &copy;  &#169; <br>
® registered trademark  &reg; &#174; <br>
      </div>
      <div>message with newlines
      next line
      last line
      </div>
    `;

Preprocessed Code

{
      'div_4': [
        'message with HTML entities &lt;\uD83D\uDE09&gt;"&amp; {1}\n  non-breaking space  &nbsp;  &nbsp; {2}\n&lt; less than &lt;  &lt; {3}\n&gt; greater than  &gt;  &gt; {4}\n&amp; ampersand &amp; &amp; {5}\n" double quotation mark "  " {6}\n\' single quotation mark (apostrophe)  \'  \' {7}\n\xA2 cent  \xA2  \xA2 {8}\n\xA3 pound \xA3 \xA3 {9}\n\xA5 yen \xA5 \xA5 {10}\n\u20AC euro  \u20AC  \u20AC {11}\n\xA9 copyright \xA9  \xA9 {12}\n\xAE registered trademark  \xAE \xAE {13} ',
        '&lt;br&gt;', // placeholder; no need to translate
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;',
        '&lt;br&gt;'
      ],
      'div_5': 'message with newlines\n      next line\n      last line '
    }

JSON

{
    "div_4": [
      "message with HTML entities <😉>\"& {1}\n  non-breaking space       {2}\n< less than <  < {3}\n> greater than  >  > {4}\n& ampersand & & {5}\n\" double quotation mark \"  \" {6}\n' single quotation mark (apostrophe)  '  ' {7}\n¢ cent  ¢  ¢ {8}\n£ pound £ £ {9}\n¥ yen ¥ ¥ {10}\n€ euro  €  € {11}\n© copyright ©  © {12}\n® registered trademark  ® ® {13} ",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>",
      "<br>"
    ]
}

XLIFF

        <source>message with HTML entities &lt;😉>"&amp; {1}
  non-breaking space       {2}
&lt; less than &lt;  &lt; {3}
> greater than  >  > {4}
&amp; ampersand &amp; &amp; {5}
" double quotation mark "  " {6}
' single quotation mark (apostrophe)  '  ' {7}
¢ cent  ¢  ¢ {8}
£ pound £ £ {9}
¥ yen ¥ ¥ {10}
€ euro  €  € {11}
© copyright ©  © {12}
® registered trademark  ® ® {13} </source>
        <target state="needs-translation">message with HTML entities &lt;😉>"&amp; {1}
  non-breaking space       {2}
&lt; less than &lt;  &lt; {3}
> greater than  >  > {4}
&amp; ampersand &amp; &amp; {5}
" double quotation mark "  " {6}
' single quotation mark (apostrophe)  '  ' {7}
¢ cent  ¢  ¢ {8}
£ pound £ £ {9}
¥ yen ¥ ¥ {10}
€ euro  €  € {11}
© copyright ©  © {12}
® registered trademark  ® ® {13} </target>
      </trans-unit>
      <trans-unit id="world-clock-container.div_4.1">
        <source>&lt;br></source>
        <target state="needs-translation">&lt;br></target><!-- placeholder; no need to translate -->
      </trans-unit>

t2ym / i18n-element

[lit-html] HTML entities in a part are shown as they are without decoding #64