search

bjsion / tongue-tied

Translation and resource management tool
1 stars 0 forks source link

Encoding not properly displayed - resaving the keyword as displayed ruins the translation #98

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Version Info
TongueTied:1.3.3
OS: Mac OS X 10.5.8
Java: 1.6.0_17
MySql: 5.1.11 UTF-8 encoded
Server: Tomcat 5.5.27

What steps will reproduce the problem?
1. Add a translation (example I use is in French - See 'Picture 5')
2. Save the translation (see 'Picture 6' for how it's properly saved in
MySQL DB)
3. View the keyword (see  'Picture 7')
4. Click on the keyword again to open up keyword details page (see 'Picture 8')
5. Resave the keyword (see 'Picture 9' for how it's improperly saved in the DB)

What is the expected output? What do you see instead?
Expect to have the translation always properly saved in the DB and properly
displayed to the user. Instead, it is improperly displayed, and then this
improper rendering gets saved to the database, ruining the translation. 

Please provide any additional information below.

I am using a UTF-8 encoded MySQL DB. Is this correct? What encoding does
TongueTied use?

Original issue reported on code.google.com by kyle.br...@gmail.com on 24 Feb 2010 at 9:57

Attachments:

GoogleCodeExporter commented 9 years ago 
FYI I have tested in Firefox 3.6 and Safari 4.0.4 with the same result.

Original comment by kyle.br...@gmail.com on 24 Feb 2010 at 10:07

GoogleCodeExporter commented 9 years ago 
So far I am not able to replicate the problem. My MySQL server was running with 
an
encoding of latin1. I will test later with utf8 encoding on my DB.

Also what charset is your JVM using? In the Tonguetied server log there is a 
line
that will tell you. It looks something like this: 
INFO [main] (ServletContextInitializer.java:142) - Charset for this JVM: UTF-8

Finally are you passing any encoding parameters as part of your jdbc url? If so 
what
are they? When I have been testing, I have not added any parameters to the jdbc 
url
for MySQL.

Original comment by bjs...@gmail.com on 24 Feb 2010 at 3:43

GoogleCodeExporter commented 9 years ago 
Thanks for trying to figure this out. 

I changed my MySQL DB to use a latin1 encoding with no fix to this problem. 

I've got tonguetied on two different machines inside Tomcat (with MySQL) with 
the
exact same behaviour. On the linux machine:  Charset for this JVM: UTF-8 
On the Mac machine: Charset for this JVM: MacRoman

I didn't have any parameters as part of my jdbc url, but then I tried adding:
jdbc:mysql://localhost:3306/tonguetied?autoReconnect=true&useUnicode=true&charac
terEncoding=UTF8&characterSetResults=UTF8
but this had no improved effect either. 

I also tried two different versions of the MySQL driver:
mysql-connector-java-5.1.6.jar and 5.1.11 with no fix either. 

I attach my tomcat log file from my Mac if this helps.

Do you know of someone that has a successful configuration using Tomcat? 

I'm guessing that you're going to have to enforce some encoding like UTF-8 in 
the
database so that you know how to write and then read the bytes in in a 
standardised
format. My guess is that the code is expecting to read a particular encoding 
when
displaying the strings, but receives a different encoding and hence displays the
wrong strings.

Original comment by kyle.br...@gmail.com on 25 Feb 2010 at 4:34

Attachments:

GoogleCodeExporter commented 9 years ago 
The problem occured because tomcat's default encoding is ISO-8859-1. In order 
to fix
the problem the body of the post needs to be passed through a filter to encode
characters. This is added in the web.xml with the following entries:

    <filter>
        <filter-name>encodingFilter</filter-name>

<filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-cla
ss>
        <init-param>
            <param-name>encoding</param-name>
            <param-value>UTF-8</param-value>
        </init-param>
        <init-param>
            <param-name>forceEncoding</param-name>
            <param-value>true</param-value>
        </init-param>
    </filter>

    <filter-mapping>
        <filter-name>encodingFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

I have attached a fixed version of the web.xml. Kyle, Could you please paste 
this in
your <TOMCAT_HOME>/webapps/tonguetied/WEB-INF directory and confirm that the 
problem
is solved.

Original comment by bjs...@gmail.com on 26 Feb 2010 at 12:48

Attachments:

GoogleCodeExporter commented 9 years ago

Original comment by bjs...@gmail.com on 26 Feb 2010 at 1:06

GoogleCodeExporter commented 9 years ago 
Fantastic. I can confirm that the problem is solved with the fixed version of 
the
web.xml you provided. Thanks for your quick response!

Original comment by kyle.br...@gmail.com on 26 Feb 2010 at 4:40

GoogleCodeExporter commented 9 years ago

Original comment by bjs...@gmail.com on 27 Feb 2010 at 12:38