search

posm / OpenMapKitServer

OpenMapKit Server is the lightweight server component of OpenMapKit that handles the collection and aggregation of OpenStreetMap and OpenDataKit data.
http://openmapkit.org
BSD 3-Clause "New" or "Revised" License
46 stars 25 forks source link

character encoding issues #61

Closed danbjoseph closed 7 years ago

danbjoseph commented 8 years ago

Special characters are garbled in the json data on the server.

_Mapper_registrationv1-0-2/44d28929-b177-4169-89db-1f1dc639895d/data.json

"FIRST_NAME": "Saa Célestin",

_Mapper_registrationv1-0-2/912774e0-005f-4ae7-9404-6b8338d1498d/data.json

"QUARTIER_OU_DISTRICT": "Kènèma",

_Mapper_registrationv1-0-2/1fae7871-0608-4c5d-9748-b3c33066b8bd/data.json

  "REGION_NAT": "Basse guinée",
hallahan commented 8 years ago

I see. The XML version is fine. Something funky with turning that XML to JSON.

http://omkserver.com/omk/data/submissions/Mapper_registration_v1-0-2/1fae7871-0608-4c5d-9748-b3c33066b8bd/data.xml

hallahan commented 8 years ago

It's not fs.writeFile, because that defaults to UTF-8

https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback

hallahan commented 8 years ago

I think the issue is in xform-to-json

https://github.com/digidem/xform-to-json

Perhaps xml2js is not doing things in utf-8?

https://github.com/Leonidas-from-XIV/node-xml2js

Requires further investigation.

mojodna commented 7 years ago

This looks like it's actually a Chromium bug: https://bugs.chromium.org/p/chromium/issues/detail?id=438464

If I use curl or download the JSON, things look copacetic.

@danbjoseph please re-open if this is causing problems and isn't just aesthetic when viewed in a browser.