RPTools / maptool

Virtual Tabletop for playing roleplaying games with remote players or face to face.
http://rptools.net
GNU Affero General Public License v3.0
787 stars 259 forks source link

[Bug]: frame5() not handling UTF-8 emojis in certain situations which work in chat and frame #4832

Open Baaaaaz opened 3 months ago

Baaaaaz commented 3 months ago

Describe the Bug

UTF-8 emojis do not display correctly in frame5 in certain situations when they do in chat and frame.

Specifically I was trying to convert some markdown which contain emojis to html, and the converted emoji is displaying as ������. Further investigation shows frame5 does not display UTF-8 emojis in other situations either.

To Reproduce

  1. Create a new macro
  2. Paste and run the MT script below
    
    <table>
    <tr><th>Case</th>                       <th>Value</th>                          <th>markdownToHtml</th></tr>
    <tr><td>0. raw</td>                     <td>🔒</td>                              <td></td></tr>
    <tr><td>1. quoted</td>                  <td>"🔒"</td>                            <td>[r: markdownToHtml("🔒")]</td></tr>
    <tr><td>2. entity code</td>             <td>&#128274;</td>                      <td>[r: markdownToHtml("&#128274;")]</td></tr>
    <tr><td>3. quoted variable</td>         <td>[r: key = "🔒"]</td>                 <td>[r: markdownToHtml(key)]</td></tr>
    <tr><td>4. entity code variable</td>    <td>[r: htmlKey = "&#128274;"]</td>     <td>[r: markdownToHtml(htmlKey)]</td></tr>
    </table>

[frame("FRAME"): {

Case Value markdownToHtml
0. raw 🔒
1. quoted "🔒" [r: markdownToHtml("🔒")]
2. entity code 🔒 [r: markdownToHtml("🔒")]
3. quoted variable [r: key = "🔒"] [r: markdownToHtml(key)]
4. entity code variable [r: htmlKey = "🔒"] [r: markdownToHtml(htmlKey)]

}]

[frame5("FRAME5"): {

Case Value markdownToHtml
0. raw 🔒
1. quoted "🔒" [r: markdownToHtml("🔒")]
2. entity code 🔒 [r: markdownToHtml("🔒")]
3. quoted variable [r: key = "🔒"] [r: markdownToHtml(key)]
4. entity code variable [r: htmlKey = "🔒"] [r: markdownToHtml(htmlKey)]

}]


3. Observe different outputs to chat/frame vs frame5 where ������ is output to frame5 in certain situations for both `🔒` and `&#128274;`

### Expected Behaviour

Consistent and correct display of UTF-8 emojis across chat/frame/frame5.

### Screenshots

![image](https://github.com/RPTools/maptool/assets/131240480/18084cb9-8152-4031-b251-d400472c3c7e)

### MapTool Info

1.14.3

### Desktop

Windows 11

### Additional Context

chat log shows `&#128274;` and `🔒` is converted to `&#55357;&#56594;`

Also affects token notes/gm notes depending which MIME type and which tab is selected, but seems to display okay in the notes pop up, but not sure if this is related:

Notes text tab:
![image](https://github.com/RPTools/maptool/assets/131240480/e39e9aed-5706-4770-ab97-a084c4454216)

Equivalent Notes HTML tab:
![image](https://github.com/RPTools/maptool/assets/131240480/ebe3dd0a-1bdb-49dd-a6eb-b1e534d6de82)

Token Notes display:
![image](https://github.com/RPTools/maptool/assets/131240480/7a52108f-de24-4878-a7dd-6323afe678ed)