Unicode symbol not rendered under very specifc circumstances

[sp1r1t]

Under certain conditions Orgzly fails to show the unicode symbol and instead displays seemingly unrelated characters.

Here is video of the bug: https://github.com/orgzly-revived/orgzly-android-revived/assets/1295554/5ed390f5-1da2-491b-b230-b73ae498746a

I stumbled upon this by accident after switching to Unicode symbols instead of the TODO/DONE strings while syncing over the filesystem. The peculiarity of this bug grabbed my attention so I did some investigation.

To summarize my findings: This happens when

• There is one Unicode character in the headlines
• The sum of regular characters ([a-z]) and headlines is greater than 30
• Spaces within a character sequence count as 1 character
• Numbers behave like spaces
• It only happens for certain Unicode characters
• It only happens when syncing (from the filesystem). When the same pattern is produced from within Orgzly, the character display is correct.

These are the tests I did: orgzly-unicod-bug-testdata.txt

This is definitely a minor bug regarding functionality, but I really wonder what is going on here.

[polaris64]

I've just come across this issue too. I haven't been able to do extensive debugging but I was using the latest version from F-Droid on a Google Pixel 8. I wasn't able to fix it so I had to install the upstream Orgzly instead and that worked fine.

[sp1r1t]

Where you able to make the unicode characters work again after changing the org file back to it's previous state (before the bug happened)?

I experienced this a for a while, then I always added some text to the note and it rendered fine again. Now I realize I didn't have the issue in months now. I'm using the latest F-Droid version.

[polaris64]

@sp1r1t no unfortunately by the time I had noticed I wasn't able to revert it. In my case I have the files synced between my phone (Orgzly) and my computer via Syncthing. I noticed the bug after I made an edit on my phone and then saw in Emacs that the file encoding had been changed (no longer UTF-8). I changed things back in Emacs but things were still rendering incorrectly in Orgzly Revived. As soon as I put the upstream Orgzly back on it rendered fine.

Perhaps it has something to do with file type auto-detection in Orgzly?

[sp1r1t]

@polaris64 I see. I have the same setup with syncthing and then doing a filesystem sync in Orgzly. Interesting that the encoding was changed in your case. I assume this is not the root cause though, as I was able to make Orgzly render the unicode character correct and incorrect by editing the file directly, as you see in my video. I used the NMM editor to do it. At least this one shows UTF-8 all the time.

As far as a I know in Revived the file scanning is not changed, so actually things just get more mysterious for now.

Can you install Revived and upstream side by side and sync the same directory? If that shows a difference I would strongly assume the bug was introduced in the Revived code base, which would narrow our search vector down drastically. Although I don't think this being the case tbh.

I think I never tested clearing the Orgzly database while having the bug and then adding the local repository again.

I've not gone through the code of how Orgzly detects file types. I'd assume it's all text files anyway. Or I guess you mean the encoding? Do you have more insights on that?

[polaris64]

@sp1r1t of course, I'll reinstall it and I'll run some tests, hopefully over this weekend. I'll report back with what I find out.