search

eclipse-archived / smarthome

Eclipse SmartHome™ project
https://www.eclipse.org/smarthome/
Eclipse Public License 2.0
862 stars 783 forks source link

Umlauts are broken after restart #6729

Open maihacke opened 5 years ago

maihacke commented 5 years ago

I have some things, which contain umlauts in their name, e.g:

Rollladen EG Büro Straße

I can enter those in PaperUI and they are correctly shown, but after a restart of openhab, they become corrupted:

Rollladen EG B?ro Stra?e

I did some checks. The umlauts are broken on my notebook (MacOS/Chrome) and mobile (iOS/Safari). Also it doesn’t matter if the umlauts where entered on the notebook or mobile. Without restarted they are alway shown correctly on both devices. I did have a look to the JSON-DB (org.eclipse.smarthome.core.thing.Thing.json). The umlauts are broken in the json (saved with ?). I also did some checks with curling the REST-API the umlauts are broken (after restart) / correct (before restore) there in the same way as in paperui. So it is not a problem in the browser or paperui.

UI is in English.

Openhab Version 2.4final

https://community.openhab.org/t/umlauts-broken-after-restart-in-oh-2-4/60167?u=maihacke

maggu2810 commented 5 years ago

I use MapDB instead of JSON DB and never hit that problem. Can you try to switch to MapDB for a moment and check if the problem exists there, too? If it is working using MapDB, we have to fix the (de)serialization on writing the JSON files (e.g. force UTF-8 encoding on read / write).

maihacke commented 5 years ago

How is it possible to switch to MapDB, I thought that is no longer an option?

maggu2810 commented 5 years ago

In general you have to remove the jsondb bundle and add the mapdb bundle. For Karaf based distributions I would expect using "feature:uninstall" and "feature:install" with the respective Karaf features should to its job. But for openHAB itself (it is no option or how it can be done correctly) you have to question e.g. the community forum.

maihacke commented 5 years ago

I did some tests. Starting the JVM with -Dfile.encoding=UTF8 solves the problem. Looking at the code in https://github.com/eclipse/smarthome/blob/master/bundles/storage/org.eclipse.smarthome.storage.json/src/main/java/org/eclipse/smarthome/storage/json/internal/JsonStorage.java the String conversion to/from bytes is a little bit "dangerous", since the getBytes-Method and instantiation of the FileReader rely on the platform encoding. Maybe it would be better to enforce UTF-8 here.

MHerbst commented 5 years ago

For openHAB on Windows there is a PR that was already merged. The patch will ensure that on Windows the encoding will default to UTF-8 (see https://github.com/openhab/openhab-distro/pull/833).

Where are you running your openHAB server?

maihacke commented 5 years ago

I’m using the debian packages on a ARM-platform

MHerbst commented 5 years ago

That' interesting. I am using openHAB on a Raspberry and it seemed to me that UTF-8 is the default encoding on this platform. Are you using the Oracle JDK or another JDK? Anyway, I think it would be best to set the encoding explicitly on all platforms to UTF-8 to avoid this kind of problems.

maihacke commented 5 years ago

I‘m using zulu-embedded java. The default encoding for java depends on the environment (locale). So yes it would be good idea to enforce UTF-8 e.g. via the startup scripts.

maggu2810 commented 5 years ago

The JSON DB should use UTF-8 encoding regardless of the system locale, environment variables, startup scripts etc.

maggu2810 commented 5 years ago

@maihacke Can you give this changes a try: https://github.com/maggu2810/smarthome/pull/new/jsondb-utf8

maihacke commented 5 years ago

Tried to compile that, but it fails with

[ERROR] Failed to execute goal org.eclipse.xtend:xtend-maven-plugin:2.14.0:compile (default) on project org.eclipse.smarthome.model.core: Execution default of goal org.eclipse.xtend:xtend-maven-plugin:2.14.0:compile failed: An API incompatibility was encountered while executing org.eclipse.xtend:xtend-maven-plugin:2.14.0:compile: java.lang.NoSuchMethodError: org.eclipse.jdt.internal.compiler.batch.FileSystem.getClasspath(Ljava/lang/String;Ljava/lang/String;ZLorg/eclipse/jdt/internal/compiler/env/AccessRuleSet;Ljava/lang/String;Ljava/util/Map;)Lorg/eclipse/jdt/internal/compiler/batch/FileSystem$Classpath;

I never compiled the smarthome project before. Is there a special java/maven version required?

maggu2810 commented 5 years ago

I will compile the branch and post a link to a bundle soon. You need a JDK version 1.8 to build ESH.

maggu2810 commented 5 years ago

org.eclipse.smarthome.storage.json-0.11.0-SNAPSHOT.jar

maihacke commented 5 years ago

I'll test it, but it will take somedays...

maggu2810 commented 5 years ago

@maihacke Have you already found some time to test?

maihacke commented 5 years ago

What whould be the best way to test that jar? Should I replace ./runtime/system/org/eclipse/smarthome/storage/org.eclipse.smarthome.storage.json/0.10.0.oh240/org.eclipse.smarthome.storage.json-0.10.0.oh240.jar with it?

cweitkamp commented 4 years ago

I think the root cause is similar to #6630 and has been fixed with https://github.com/openhab/openhab-distro/pull/833. This issue can be closed.

maihacke commented 4 years ago

https://github.com/openhab/openhab-distro/pull/833 changes encoding for windows platform in setenv.bat. Since I do not use that file on linux, this cannot fix this problem here.