Allow a different character set to be set when running tests

[GoogleCodeExporter]

I'd like to be able to specify something like:
     java -jar JsTestDriver-1.2.jar --charset "utf-8" --testOutput test-results   --tests all
and have the system assume everything is done in the specified character set.

The reason for this is that I do a lot of work with non-ascii characters.  in 
theory I could have things 
like \u00fc scattered through my code, but given the amount of such characters 
in some code, it is 
much nicer to be able to see the character directly.  At the moment, however, a 
utf8 character in my 
file seems to get munged by the time the test is run.  Providing a flag to set 
a global character set 
would allow me to mimic the environment all my code will run in (one where 
every html file is 
specifying the character set explicitly).

Original issue reported on code.google.com by davidjoh...@gmail.com on 24 Dec 2009 at 11:44

[GoogleCodeExporter]

Enclosed is a patch for a preliminary fix for this issue.  Details about using 
this below, in case you (like me) are 
being blocked by this issue.

TO USE THIS:
Add a line like the following to your config file:
charset: UTF-8
Any character set name that Java understands and your browser will understand 
can be used.  When specified, 
this assumes all files listed in the file are encoded with that character 
encoding

TO APPLY THE PATCH FILE
If you unjar the JsTestDriver-1.2-src.jar, name the directory the com and 
META-INF directories go into 
"originalSource".  Then run
patch -p0 < charset.patch 
and this will modify the source to this modification. You can then build this 
source and replace the old version in 
the main JsTestDriver distribution (you'll need to make sure you get runner.js, 
heartbeatclient.js and 
jstestdrivernamespace.js from the main distribution and add them to the new 
version of the source)

KNOWN ISSUES
- If you change the charset in your .config file without restarting the server, 
the new files won't be delivered
- The xml data generated by the test is not in unicode.
- I've not yet tested what happens if you try to put text from your test into 
the HTML document the test is 
running in.  
- There's no way to specify a character set per file (not sure this is 
important)

Original comment by davidjoh...@gmail.com on 9 Jan 2010 at 4:26

Attachments:

• charset.patch

[GoogleCodeExporter]

Original comment by corbinrs...@gmail.com on 10 Jan 2010 at 9:32

• Changed state: Accepted

[GoogleCodeExporter]

Enclosed are two simple test situations for this.  both have all the files in 
UTF8.  The one labeled deugUSASCII 
has the charset set to US-ASCII, and it won't work right (one of the tests will 
fail, one will work.  the one that 
works does so only accidentally).  The one labeled debugUTF8 will work.  There 
need to be other test cases, but I 
need to do more investigation into what other character sets are supported in 
Java (much less the browsers).

Original comment by davidjoh...@gmail.com on 18 Jan 2010 at 8:54

Attachments:

• debug.zip

[GoogleCodeExporter]

+1. I have hard-patched utf-8 in ugly way in my checkout and it seems to me 
that only reasonally good solution is to rewrite all handling to use byte 
arrays instead of strings. For example, fixing encoding at loading does not 
resolve problems (and if we want to serve binary content it makes it even 
worse) as HtmlDocParser somewhat mangles the utf-8 encoding anyway. So I think 
the only reliable way for encoding to go through loading/processing/sending 
request/processing/downloading to browser cycle is to have byte arrays for all 
stages except the last and command flag which specifies the proper charset 
which will be sent to browser.

Original comment by ppersh...@gmail.com on 12 Jan 2011 at 9:52

[GoogleCodeExporter]

+1.  This is a dealbreaker for me, as Visual Studio encodes all JavaScript 
files with the UTF-8 BOM by default 
(http://connect.microsoft.com/VisualStudio/feedback/details/321798/default-or-un
icode-file-save-encoding).  So my only alternatives are to patch the source or 
batch de-BOM all my existing JavaScript files and perform an extra save step 
when creating subsequent JS files.

Original comment by objec...@gmail.com on 28 Mar 2011 at 8:21

[GoogleCodeExporter]

Fix note for dealing with MS non-standard utf-8:
http://www.rgagnon.com/javadetails/java-handle-utf8-file-with-bom.html

Original comment by corbinrs...@gmail.com on 28 Mar 2011 at 9:16

[GoogleCodeExporter]

Issue 210 has been merged into this issue.

Original comment by corbinrs...@gmail.com on 30 Mar 2011 at 10:59

[GoogleCodeExporter]

+1 I'm having the same problem which is a major bug as far as I am concerned. 
Not being able to specify the charset is really painful !!

Any news regarding this ?
Do you have a valid fix or do you need any help (more investigation, other 
patch) ?
Any plan to integrate this in the next release ? (and if so, when?)

Original comment by olivier....@gmail.com on 5 Aug 2011 at 2:07

[GoogleCodeExporter]

Not the next release, as I'm finishing it up, and it really needs to go out.

The issue requires some pretty vast changes to properly deal with charsets, so 
the release after seems likely.

Original comment by corbinrs...@gmail.com on 5 Aug 2011 at 2:55

[GoogleCodeExporter]

:(
Ok, thank your for your answer.
Can you keep us in touch through comments once your have a fix in the SVN ? 
I'll try to test it with our JS to give you another confirmation it work as 
expected in misc environments .

Original comment by olivier....@gmail.com on 5 Aug 2011 at 3:04

[GoogleCodeExporter]

Attached is a patch I made for adding binary file support.  It doesn't add 
additional charset support, but it certainly makes it much easier to do so.  If 
someone is willing to add charset support (maybe by merging David's original 
patch), the conversion should only have to be done in 
FileInfo.getDataAsString().

The patch applies to r1052.

Original comment by dor...@gmail.com on 3 Oct 2011 at 7:07

Attachments:

• jstd_binary_file_fix.patch

[GoogleCodeExporter]

I'd like to vote for this issue too. Even support for UTF-8 only will be a 
major step forward. E.g. why don't you hard-code UTF-8 instead of plain ASCII? 
Since UTF-8 is compatible with ASCII it shall not be a regression given the 
current behavior.

Original comment by pavel.ge...@gmail.com on 25 Oct 2011 at 1:39

[GoogleCodeExporter]

I completely agree with pavel on this suggestion. ASCII is so 1990 ! :)

Original comment by olivier....@gmail.com on 25 Oct 2011 at 1:41

[GoogleCodeExporter]

Yes, please this is a must. I'm using YUI and some of their files contain UTF8 
chars as part of some regular expressions. Currently I have to remove all the 
UTF8 chars to run the tests with js test driver which is very ugly.

Original comment by giuliano...@gmail.com on 17 Feb 2012 at 3:37

[GoogleCodeExporter]

I also agree that it would be great to get this fix into the next version.  Any 
word on when that may be?  Many thanks!

Original comment by mull...@gmail.com on 26 Jun 2012 at 9:14

[GoogleCodeExporter]

I also have tests written in UTF-8. Is there work being done on this issue? 
Anyone has a patch fresher than the one posted above for v1.2? 

Original comment by cyb...@gmail.com on 8 Aug 2012 at 11:38

[GoogleCodeExporter]

We also got bit by this, d3.js is becoming a popular library and it uses the 
characters for mu and sigma. This is badly needed.

Original comment by mbaker....@gmail.com on 10 Aug 2012 at 6:50

[GoogleCodeExporter]

+1!!! We have a lot of UTF8 stuff in our project, especially dealing with the 
word stemming in different languages. So I'd like to vote for this issue.

Original comment by mma...@hwdtech.ru on 3 Oct 2012 at 1:23

[GoogleCodeExporter]

We also need a fix for this.
We have js code that parse texts in different languages and we want to test it, 
so UTF-8 is badly needed.
We tried to find simple fix this problem but did not succeed.

Original comment by menshov.a.v@gmail.com on 3 Oct 2012 at 1:24

[GoogleCodeExporter]

Very important issue. Our leadership demands features on Russian.

Original comment by Lilo.Ja...@gmail.com on 3 Oct 2012 at 2:48

[GoogleCodeExporter]

Hello all. here's a patch I've made which can make the 1.3.5 version of 
js-test-driver handle UTF-8 files.
You can see the code changes in the enclosed .diff file.  Alternately, I've 
included .class files for the relevant changes, so you can follow the steps in 
the README.txt to build your own patched js-test-driver without going through 
the hassle of setting up a whole dev environment around it.

This is contributed to the community with the usual disclaimer of I offer no 
warranties etc. for your experience with this.  But, I still hope it is useful.

Original comment by bie...@gmail.com on 8 Oct 2012 at 5:14

Attachments:

• patch-for-1.3.5.zip

[GoogleCodeExporter]

Thanks, man! It works as expected.

Original comment by mma...@hwdtech.ru on 8 Oct 2012 at 5:41

[GoogleCodeExporter]

Thank you very much! It works! I tried fix this yesterday but did not find all 
problem parts 

Original comment by jl...@hwdtech.ru on 8 Oct 2012 at 5:41

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Here is another patch based on the patch provided by biede0 in comment 21.
Patched version of JsTestDriver-1.3.5.jar is available for download: 
http://git.jetbrains.org/?p=idea/contrib.git;a=blob;f=JsTestDriver/lib/jstestdri
ver-core/JsTestDriver-1.3.5-patched.jar;h=08e615f13dbfe7953ae1ee14472251bf637f2c
a8;hb=HEAD.
The only improvement is that messages sent to console.log() are also encoded in 
UTF-8.
http://confluence.jetbrains.com/display/WI/Patched+JsTestDriver-1.3.5

Original comment by Sergey.S...@gmail.com on 12 Oct 2012 at 5:47

Attachments:

• jstd-1.3.5-issue-85-fix.patch

[GoogleCodeExporter]

Thanks Sergey!  That's fantastic!

Original comment by bie...@gmail.com on 16 Oct 2012 at 3:50

[GoogleCodeExporter]

Is this patch going to be released as properly versioned release any time?

Original comment by scott.ca...@sap.com on 23 Jul 2013 at 7:48