search

akvo / akvo-flow-mobile

Akvo Flow app
GNU General Public License v3.0
18 stars 16 forks source link

Devices with special fonts show as ????? in dataset #1762

Closed janagombitova closed 3 years ago

janagombitova commented 3 years ago

Describe the bug On WHH we received a question about why some data in the dataset is shown as ??????????. This was for free text submissions. But also for Device ID. The WHH team then noticed that the 2 devices with this issue show in a special font in the Flow online workspace.

Screen Shot 2021-01-07 at 14 50 44 Screen Shot 2021-01-07 at 14 51 12

The questions are (might need to split into separate issues, but I am trying to add everything here so we have a base):

To Reproduce Steps to reproduce the behavior:

  1. Go to 'welthungerhilfe.akvoflow.org'
  2. Check the Devices tab and use the browser search option to find the 2 devices
  3. Dataset: Syria_Turkey_Lebanon>2020>SYR1078>SYR1078_Agricultural_HOH_Assessment_form has the issue on the dataset level

Support issue https://akvoo.reamaze.com/admin/conversations/typo-in-akvo-flow-data

valllllll2000 commented 3 years ago

I am still investigating what happened but I have looked at the data.json file and instead of the ??? here is the text: 𝗮𝗵𝗺𝗲𝗱 𝗱𝘆𝗮𝗯 𝗮𝗹𝗮𝗯𝗼𝗱

if I paste it into the IDE code it becomes: String t = "\uD835\uDDEE\uD835\uDDF5\uD835\uDDFA\uD835\uDDF2\uD835\uDDF1 \uD835\uDDF1\uD835\uDE06\uD835\uDDEE\uD835\uDDEF \uD835\uDDEE\uD835\uDDF9\uD835\uDDEE\uD835\uDDEF\uD835\uDDFC\uD835\uDDF1";

but java code is able to read it correctly. In the datastore, I can find this string and it's all correct.

I think the dashboard is not able to display these characters because it reads them with the wrong encoding but the data is all there and correct. When we export the sheet, the characters are encoded wrong but if I paste that word in the spreadsheet myself, it is displayed correctly.

We would need to look at the dashboard code on how data is displayed and also in the code which generates spreadsheets.

I really have no idea how they generated these characters, could they have done a copy paste? Not sure if some changes in the app are possible/doable to convert this text into normal text.

valllllll2000 commented 3 years ago

After more research, I have come to this conclusion, for example the letter 'a' is represented: https://www.fileformat.info/info/unicode/char/1d5ee/index.htm so it's \uD835\uDDEE for the letter but if you remove \uD835 then the code does not work and the letter cannot be read

valllllll2000 commented 3 years ago

this has been solved in akvo/akvo-flow#3716