[Messages/UI ?] Parts of messages missing

[thyttan]

Affected hardware version

Bangle 2

Your firmware version

2v19.23

The bug

Message contents arrive on watch with parts missing.

I have noticed this for some time. Also mentioned on https://github.com/espruino/BangleApps/issues/3033#issuecomment-1737900594.

Expected behavior

The whole message should arrive complete.

Steps to reproduce

Unknown.

Additional info

A message from my news app arrived on the watch but it was incomplete:

The message as it's presented in the news app:

I tested to send the complete message body with the Gadgetbridge debug menu and every letter arrived on the watch complete:

Gadgetbridge

Built from source with changes in this PR: https://codeberg.org/Freeyourgadget/Gadgetbridge/pulls/3151

Bangle paired as companion device.

Device Specific Settings

• Text as Bitmaps : Off
• Transliteration : Not set.
• Allow hig MTU : On
• Allow Internet Access : On
• Allow Intents : On

Installed apps

sched (0.22) messageicons (0.05) widmsggrid (0.05) notify (0.13) torch (0.11) calculator (0.07) widram (0.03) widbthide (0.01) smpltmr (0.08) clkinfostopw (0.03) runplus (0.21) agpsdata (0.06) messagesmusic (0.05) quicklaunch (0.15) hrmmar (0.02) kbmulti (0.07) podadrem (0.09) clock_info (0.07) dtlaunch (0.24) recorder (0.30) sleeplog (0.15) backswipe (0.02) boot (0.59) setting (0.63) slidertest (0.01) multitimer (0.07) sixths (0.01) widanclk (0.03) health (0.27) edgeclk (0.03) rebbleagenda (0.01) fastload (0.06) fastreset (0.02) lightswitch (0.09) messagegui (0.74) messages (0.62) spotrem (0.11) alarm (0.43) widminbate (0.04) mysetup (0.01) android (0.33) swscroll (0.05) oxofocus (0.02)

[gfwilliams]

It's probably not a complete surprise the missing chars start after ö which is in the UTF8 codepoint range (246). However I thought Gadgetbridge should have converted that with escape chars: https://codeberg.org/Freeyourgadget/Gadgetbridge/src/branch/master/app/src/main/java/nodomain/freeyourgadget/gadgetbridge/service/devices/banglejs/BangleJSDeviceSupport.java#L389-L390

As you mentioned in that last comment, Text as Bitmaps should fix this though and it is the recommended setting.

Please could you attach a log dump from Gadgetbridge for when this happened? It'll let me see what was sent to the watch and to then reproduce/fix it.

I guess it's possible that there was an actual unicode char (>255) which got encoded as \uxxxx after the other stuff - that'll cause Espruino to convert the existing non-unicode string to unicode, and maybe there is some kind of issue there - but I'd need to know the full string to be sure

[thyttan]

Please could you attach a log dump from Gadgetbridge for when this happened?

Turned on logging now but didn't have it on before. Hopefully I get the bug again soon.

[thyttan]

Back with a report on fw2v21.6. Text as bitmap was NOT turned on.

Gadgetbridge log reports this (abbreviated):

13:03:40.819 [main] DEBUG n.f.g.e.NotificationListener - Notification 12130341 posted: packageName=se.svt.svti.android.nyhetsapp, priority=0, category=null
13:03:40.993 [main] INFO  n.f.g.e.NotificationListener - Processing notification 1707492737, age: 532, source: se.svt.svti.android.nyhetsapp, flags: 16
13:03:41.010 [main] DEBUG n.f.g.e.NotificationListener - No Notification Filter found
13:03:41.032 [main] DEBUG n.f.g.s.DeviceCommunicationService - Service startcommand: nodomain.freeyourgadget.gadgetbridge.devices.action.notification
13:03:41.054 [main] INFO  n.f.g.s.d.b.BangleJSDeviceSupport - UART TX: GB({"t":"notify","id":1707492737,"src":"SVT Nyheter","title":"JUST NU","subject":"","body":"Stort eldklot sl\xe5r ut \u2013 h\xe4r exploderar badhuset vid Liseberg \u2022 Fyra personer lindrigt skadade, enligt polisen","sender":""})

Log from "Fetch device Debug Logs" in Gadgetbridge (abbreviated):

================================================
ACTION_DEVICE_CHANGED Connected 2024-02-12 01:01:30
================================================

================================================
SENDING GB({"t":"notify","id":1707492737,"src":"SVT Nyheter","title":"JUST NU","subject":"","body":"Stort eldklot sl\xe5r ut \u2013 h\xe4r exploderar badhuset vid Liseberg \u2022 Fyra personer lindrigt skadade, enligt polisen","sender":""})

================================================

{"t":"act","stp":81,"hrm":0,"mov":492}

{"t":"status","bat":29,"chg":0}

Sending

GB({"t":"notify","id":1707492737,"src":"SVT Nyheter","title":"JUST NU","subject":"","body":"Stort eldklot sl\xe5r ut \u2013 h\xe4r exploderar badhuset vid Liseberg \u2022 Fyra personer lindrigt skadade, enligt polisen","sender":""})

to my watch through the Web IDE reproduces the slightly weird off-center formatting, missing — and •. Removing \u2013 and \u2022 (corresponding to the — and •) makes the message display centered.

I'll send a more complete Gadgetbridge log to @gfwilliams privately on the forum).

[thyttan]

If I send this

— long dash around —
• bullet around •

from the Gadgetbridge debug screen I get this

if Text as bitmaps is off, and this

if Text as bitmaps is on.

Gadgetbridge log (abbreviated):

22:42:06.225 [main] INFO  n.f.g.s.d.b.BangleJSDeviceSupport - UART TX: GB({"t":"notify","id":1707761835,"src":"Bangle.js Gadgetbridge","title":"","subject":atob("PyBsb25nIGRhc2ggYXJvdW5kID8KPyBidWxsZXQgYXJvdW5kID8="),"body":atob("PyBsb25nIGRhc2ggYXJvdW5kID8KPyBidWxsZXQgYXJvdW5kID8="),"sender":atob("PyBsb25nIGRhc2ggYXJvdW5kID8KPyBidWxsZXQgYXJvdW5kID8="),"tel":atob("PyBsb25nIGRhc2ggYXJvdW5kID8KPyBidWxsZXQgYXJvdW5kID8=")})

==== EDIT ====

And if I send this

- short dash around -
— long dash around —
• bullet around •

from the Gadgetbridge debug screen I get this

if Text as bitmaps is off, and this

if Text as bitmaps is on.

Gadgetbridge log (abbreviated):

22:41:56.028 [main] INFO  n.f.g.s.d.b.BangleJSDeviceSupport - UART TX: GB({"t":"notify","id":1707761834,"src":"Bangle.js Gadgetbridge","title":"","subject":"- short dash around -\n\u2014 long dash around \u2014\n\u2022 bullet around \u2022","body":"- short dash around -\n\u2014 long dash around \u2014\n\u2022 bullet around \u2022","sender":"- short dash around -\n\u2014 long dash aro...","tel":"- short dash around -\n\u2014 long dash around \u2014\n\u2022 bullet around \u2022"})

[thyttan]

Trying to recreate the original bug on fw2v21.6:

Sending from Gadgbetbridge debug screen:

Talmannen Kevin McCarthy avsätts — partikamrater avgjorde
Första gången i USA:s historia • Republikanen förlorar posten i representanthuset

Results in:

Gadgetbridge log (abbreviated):

20:47:44.829 [main] INFO  n.f.g.s.d.b.BangleJSDeviceSupport - UART TX: GB({"t":"notify","id":1707761818,"src":"Bangle.js Gadgetbridge","title":"","subject":"Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA...","body":"Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia \u2022 Republikanen förlorar posten i representanthuset","sender":"Talmannen Kevin McCarthy avs\xe4tts \u2014 pa...","tel":"Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia \u2022 Republikanen förlorar posten i representanthuset"})

Removing the — and • characters

Sending from Gadgbetbridge debug screen:

Talmannen Kevin McCarthy avsätts  partikamrater avgjorde
Första gången i USA:s historia  Republikanen förlorar posten i representanthuset

Results in:

Gadgetbridge log (abbreviated):

22:08:54.321 [main] INFO  n.f.g.s.d.b.BangleJSDeviceSupport - UART TX: GB({"t":"notify","id":1707761827,"src":"Bangle.js Gadgetbridge","title":"","subject":"Talmannen Kevin McCarthy avs\xe4tts  partikamrater avgjorde\nFörsta g\xe5ngen i USA:...","body":"Talmannen Kevin McCarthy avs\xe4tts  partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia  Republikanen förlorar posten i representanthuset","sender":"Talmannen Kevin McCarthy avs\xe4tts  par...","tel":"Talmannen Kevin McCarthy avs\xe4tts  partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia  Republikanen förlorar posten i representanthuset"})

Observations

It seems curious to me that å and ä is replaced, with \xe5 and \xe4, while ö is not. Demonstrably it doesn't need to be as shown in the second example where — and • were omitted and the ö character shows on the watch anyway.

But ö together with — and • might be problematic by the look of it?

The ÿ characters appearing on the watch is strange.

— and • seem to contribute to uncentered formatting here also.

[gfwilliams]

So, so we're saying that in the simple dashes case it all looks fine now?

The escaping itself is I believe just what I found (more or less by trial and error) would make its way through to Espruino when sent via Gadgetbridge. The issue is as soon as you have something that's not 0-255, Espruino treats the string as UTF8, which is handled a bit differently.

It looks like there's still some issue with g.wrapString and UTF8 so this is something that'll need sorting in a firmware update:

g.setFont("6x8:2").wrapString("Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia \u2022 Republikanen förlorar posten i representanthuset",160)

[thyttan]

So, so we're saying that in the simple dashes case it all looks fine now?

Sorry, I don't now for certain - but yes, I think so by the look of the test with - short dash around -.

[gfwilliams]

Hopefully fixed now! Seems to render ok, although there is something odd going on with the x offset of lines in the message.

Overall it's miles better though, and I also fixed the issue with USA:s getting a space added after it.

[thyttan]

Thanks! I'll update my watch 🙂

[thyttan]

Sorry :P, on 2v21.14 I get:

>g.setFont("6x8:2").wrapString("Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia \u2022 Republikanen förlorar posten i representanthuset",160)
=[
  "Talmannen",
  "Kevin",
  "McCarthy",
  "avs\u00E4tts \u2014",
  "partikamrater",
  "avgjorde",
  "F\uDA8B\uDCF4a g\u00E5ngen i",
  "USA:s historia",
  "\u2022Republikanen",
  "f\uDA8A\uDF2Frar posten i",
  "representanth",
  "uset"
 ]
>GB({"t":"notify","id":1707492737,"src":"SVT Nyheter","title":"JUST NU","subject":"","body":"Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia \u2022 Republikanen förlorar posten i representanthuset","sender":""})
=undefined
>g.dump()

So it's definitely better, the ÿ's are gone for me too. But ö seems to mess with something. Long dash and bullet point are omitted (which is maybe the expected behavior?) but takes up space on the screen still, so I think that's what makes it look off center for some lines.

[thyttan]

So it's definitely better, the ÿ's are gone for me too. But ö seems to mess with something. Long dash and bullet point are omitted (which is maybe the expected behavior?) but takes up space on the screen still, so I think that's what makes it look off center for some lines.

... If I remove long dash and bullet point:

>g.setFont("6x8:2").wrapString("Talmannen Kevin McCarthy avs\xe4tts partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia Republikanen förlorar posten i representanthuset",160)
=[
  "Talmannen",
  "Kevin",
  "McCarthy",
  "avs\xE4tts",
  "partikamrater",
  "avgjorde",
  "F\xF6rsta g\xE5ngen",
  "i USA:s",
  "historia",
  "Republikanen",
  "f\xF6rlorar",
  "posten i",
  "representanth",
  "uset"
 ]
>GB({"t":"notify","id":1707492737,"src":"SVT Nyheter","title":"JUST NU","subject":"","body":"Talmannen Kevin McCarthy avs\xe4tts partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia Republikanen förlorar posten i representanthuset","sender":""})
=undefined
>g.dump()

So if the long dash and bullet point is not sent ö doesn't act up.

[gfwilliams]

Hmm, well that's interesting. So on desktop I don't get the error, and it's obviously at least a little to do with how the non-ascii ö is translated to bytes when it's sent to Espruino via the IDE.

For instance on the Bangle:

>a="Talmannen Kevin McCarthy avs\xe4tts \u2014 partikamrater avgjorde\nFörsta g\xe5ngen i USA:s historia \u2022 Republikanen förlorar posten i representanthuset";
="Talmannen Kevin M" ... "ar posten i repre"
>g.setFont("6x8:2").wrapString(a,160)
=[
...
  "F\uDA8B\uDCF4a g\u00E5ngen i",
...
 ]
>print(E.toJS(a))
"Talmannen Kevin McCarthy avs\u00E4tts \u2014 partikamrater avgjorde\nF\uDA8B\uDCF4a g\u00E5ngen i USA:s historia \u2022 Republikanen f\uDA8A\uDF2Frar posten i representanthuset"

So it's the actual raw string that has the issue in this case, not wrapString which is now doing what you might expect.

Turns out valid UTF8 chars stop at 0xF4, and ö ox 0xF6 - however some places in the code we weren't checking properly and were still treating 0xF6 as a UTF8 char.

It should be sorted now.

[thyttan]

Awesome, thanks!

[thyttan]

@gfwilliams just so I'm on the same page, we don't expect the long dashes or bullet points from above to be drawn when sent to the watch. Is that correct?

[gfwilliams]

Correct, yes - the built-in font only handles char codes 0..255 so they won't show. However when rendering PBF font files it can handle thousands of extra characters - it's something I have plans to introduce at some point.

If you enable the 'text as bitmaps' they should be handled.

And I believe Gadgetbridge actually has an option for 'transliteration' where it chooses similar ASCII chars to the unicode ones and uses them instead

[thyttan]

Thanks!