Http Response Encoding Error

[GoogleCodeExporter]

What steps will reproduce the problem?
1. In any Controller source file, encoded in utf-8, put a flash message which 
contains an accented letter such as "è": for example:

  flash("message","combinazione di email e password non è valida");

2. Run the controller and examine the message in the resulting output page

3. Try various options such as:
  - use setEncoding("UTF-8") in controller action
  - override getEncoding() method with:
      public String getEncoding() { 
        logDebug("in getEncoding");
        return("UTF-8");
      }
    (this method is not called at runtime, by the way)

What is the expected output? What do you see instead?

Expected output: "combinazione di email e password non è valida"
Actual output:   "combinazione di email e password non è valida"

What version of the product are you using? On what operating system?
Version 1.6 of Activeweb

It does not work on a Linux Opensuse OS with default encoding set to UTF-8
It does not work on a Windows 7 workstation with default encoding set to Cp1252
It appears to work on Mac OS 

The impression we get is that the setEncoding("utf-8") method DOES change the 
content-type HTTP header to: "Content-Type:text/html; charset=utf-8", BUT the 
actual byte stream in the response is nonetheless being written in another 
encoding.

It is vital for a web application in western europe to have the capacity to 
generate a response stream which is coherent with the http header of the 
response.

The Java compiler we are using is a 1.7 version, which is run via the maven 
plugin thus configured:

            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                    <encoding>${project.build.sourceEncoding}</encoding>
                </configuration>
            </plugin>

We also have:

    <properties>
         ......
        <activeweb.version>1.6</activeweb.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

We have also tried the following in the web.xml:

    <filter>
        <filter-name>dispatcher</filter-name>
        <filter-class>org.javalite.activeweb.RequestDispatcher</filter-class>
        <init-param>
            <param-name>exclusions</param-name>
            <param-value>css,img,js,favicon.ico</param-value>
        </init-param>
        <init-param>
            <param-name>root_controller</param-name>
            <param-value>home</param-value>
        </init-param>
        <init-param>
            <param-name>encoding</param-name>
            <param-value>UTF-8</param-value>
        </init-param>
    </filter>

At this point, we need assistance. 

It would seem to be an activeweb issue since we have done several tests to make 
sure it is not a compilation problem.

Original issue reported on code.google.com by jayse...@gmail.com on 21 Jun 2013 at 11:39

[GoogleCodeExporter]

upgrade to version 1.9 and try again

Original comment by i...@polevoy.org on 22 Jun 2013 at 4:04

• Changed state: WontFix

[GoogleCodeExporter]

I have upgraded to 1.9: the problem persists.

To reproduce the problem, you need an MS/Windows workstation. My colleague 
using a Mac is working on the same code, and the problem does not occur with 
him (on exactly the same project and source code).

I have checked that it is not a compilation issue. (I am using java 1.7 on 
windows).

The logDebug("combinazione di email e password non è valida"); appears 
correctly on a UTF-8 terminal window (ie: a "putty" terminal with character 
encoding set to UTF-8), but the flash message on the web page always appears 
as: "combinazione di email e password non è valida"

In servlets I manage these encoding issues with 
response.setCharacterEncoding("UTF-8").. but with activeweb I have tried dozens 
of techniques with no result.

I have also added: getConfiguration().setDefaultEncoding("UTF-8"); to the 
FreeMarkerConfig.java program.

Interesting that Freemarker is managing UTF-8 encoded templates correctly (ie: 
if I put the string "àààà" onto the template, it is rendered correctly on 
the web page).

This is strange... if the stream is not utf-8 encoded.. I would expect the 
template text to also be "scrambled".. could it be a double error producing a 
correct result?

Original comment by jayse...@gmail.com on 22 Jun 2013 at 5:50

[GoogleCodeExporter]

I do not have windows. Can you try cygwin instead of putty and see if this 
fixes the problem?

Original comment by i...@polevoy.org on 23 Jun 2013 at 5:48

[GoogleCodeExporter]

Hi Igor,

sorry, I am beginning to understand... It is actually a compilation problem. I 
wrote this controller:

package app.controllers.priv.gp;
import org.javalite.activeweb.AppController;
public class PippaController extends AppController {
        public void index() {
                logDebug("èèè" + "èèè".length());
        }
}

The string that I see on my terminal output is "èèè6".

The final 6 is not correct, should be 3, which means that the 
maven-compiler-plugin is not aware of the fact that the source java file is 
encoded in UTF-8. If that were the case I would expect the terminal output to 
be: "èèè3". The reason that the "èèè" part comes out correctly on the 
terminal is that the error in compilation is cancelled out when the string is 
reconverted into a byte stream for output. Two errors cancelling each other out 
= correct result.

So this is not an activeweb problem. It looks to me like a maven problem.

It is weird because I tell the maven compiler plugin that the source files are 
encoded in utf-8.

Even more weird is that I did a HelloWorld type example in the same project, in 
which the compiler respects the utf-8 encoding:

package app.batch;
import java.io.PrintStream;
import java.io.UnsupportedEncodingException;
public class TestUtf8 {
    /**
     * @param args
     * @throws UnsupportedEncodingException 
     */
    public static void main(String[] args) throws UnsupportedEncodingException {
        PrintStream out = new PrintStream(System.out, true, "UTF-8");
        out.println("Char: è " + "è".length());
    }
}

The output of this is:

Char: è 1

Hence, since the length of "è" is 1, this source file, which is also UTF-8 
encoded is compiled correctly under the same "pom.xml".

I am stumped as to why a source file in the "app.batch" package should be 
compiled correctly and another in the "app.controllers.priv.gp" package not.

Thanks,

JC

Original comment by jayse...@gmail.com on 24 Jun 2013 at 7:08

[GoogleCodeExporter]

OK, I think I may help. If you are running in the debug mode, then controllers 
are recompiled and reloaded by the framework on the fly. This means that the 
internal compiler settings will overwrite your maven compiler settings. 
Please, see here:
https://github.com/javalite/activeweb/blob/master/activeweb/src/main/java/org/ja
valite/activeweb/DynamicClassFactory.java#L80

So.. what to do? You can pull down code, and play with settings in the 
framework, and then you can send a pull request.
Alternatively, you can turn dynamic recompile/reload off by removing 
auto_reload flag from pom.xml. 
Either of these steps will steer you in the right direction

Original comment by i...@polevoy.org on 25 Jun 2013 at 3:49

[GoogleCodeExporter]

in the previous comment, please read "in the development mode", not "debug mode"

Original comment by i...@polevoy.org on 25 Jun 2013 at 4:34

[GoogleCodeExporter]

Thanks Igor,
when I have a moment I will look at the code. Presumably we will need to add a 
"source_file_encoding" option somewhere so that the "-encoding " flag can be 
passed to the dynamic compiler.

If so, should this "source_file_encoding" option be in the pom.xml? (But what 
if they are not using Maven?)

Original comment by jayse...@gmail.com on 25 Jun 2013 at 7:52