Error encoding JS files with UTF-8 encoding

Catmacey commented 11 years ago

Hi, I am having problems when trying to minify JS files that contain UTF-8 using JSMin from your combine.jar.

I've created a standalone version of your compressJsWithJSMin() method rather than the whole combine.cfc.

As a simple test I have the string "This is a test Франция" which results in the output "This is a test $@0=F8O"

I've tried setting a Charset string on the output toString() method but it makes no difference.

I realise that this isn't strictly an issue with your combine package but I hope you can forgive me for asking for help here.

I have a simple standalone example that demonstrates the issue;

<cfcontent type="text/html; charset=utf-8" />
<cfprocessingdirective pageEncoding="utf-8"/>
<html>
    <head>
        <title>JSMin UTF-8 test</title>
    </head>
    <body>

<cftry>
    <cfscript>
        testString = "This is a test Франция";
        joInput = createObject("java","java.io.StringReader").init(testString);
        joOutput = createObject("java","java.io.ByteArrayOutputStream").init();
        javaloader  = createObject('component','components.javaloader.JavaLoader').init([application.settings.localdir & "components/jsmin/combine.jar"]);
        jJSMin = javaloader.create("com.magnoliabox.jsmin.JSMin").init(joInput, joOutput);
        jJSMin.jsmin();
    </cfscript>

<cfoutput><pre>
TestString:"#testString#"
Native:"#joOutput.toString()#"
UTF-8:"#joOutput.toString("UTF-8")#"
ISO-8859-1:"#joOutput.toString("ISO-8859-1")#"
US-ASCII:"#joOutput.toString("US-ASCII")#"
</pre></cfoutput>

    <cfset joInput.close() />
    <cfset joOutput.close() />

    <cfcatch>
        <cfdump var="#cfcatch#" />
    </cfcatch>
</cftry>

</body>
</html>

zefer commented 11 years ago

Hi @Catmacey.

Firstly, I haven't looked at this project for so long, the code looks totally alien to me, too, as I haven't worked with CFML for 4+ years! So I'm very sorry to say I can't support this project any more. However, I do have a couple of suggestions:

Make sure the file is saved as a UTF-8 file, to ensure your testString isn't affected by encoding issues
Check your test string is output correctly before processing (which your output does, but I couldn't tell from your description if you had checked this)
Try to isolate which process is messing with the encoding, maybe you could bypass the jsmin step to see if that's the cause?

Catmacey commented 11 years ago

Hi Joe,

I thought it was a longshot to ask you for help. I could see that the project hadn't had much activity for a while.

I've tried isolating all I can and dumping the output at various stages but ultimatly I think it comes down to JSMin java.

Thanks for the response and also for the tips. I think I'll put this to one side for the moment and see if I can find another solution.

Best regards

Matt Casey

-----Original Message----- From: Joe Roberts [mailto:notifications@github.com] Sent: 11 June 2013 14:26 To: zefer/Combine Cc: Catmacey Subject: Re: [Combine] Error encoding JS files with UTF-8 encoding (#4)

Hi @Catmacey https://github.com/Catmacey .

Firstly, I haven't looked at this project for so long, the code looks totally alien to me, too, as I haven't worked with CFML for 4+ years! So I'm very sorry to say I can't support this project any more. However, I do have a couple of suggestions:

Make sure the file is saved as a UTF-8 file, to ensure your testString isn't affected by encoding issues
Check your test string is output correctly before processing (which your output does, but I couldn't tell from your description if you had checked this)
Try to isolate which process is messing with the encoding, maybe you could bypass the jsmin step to see if that's the cause?

— Reply to this email directly or view it on GitHub https://github.com/zefer/Combine/issues/4#issuecomment-19258215 . https://github.com/notifications/beacon/BtoZ-o_OF9FU1KIu2BCsma_d9hW68rnoNG4K9-B6TVC1xAFOfj9kDcY7spqx1b-5.gif

zefer commented 11 years ago

Hi Matt.

The JSmin source is available here https://github.com/zefer/Combine/blob/master/com/magnoliabox/jsmin/JSMin.java - so you could isolate it further by picking the classes it uses, if you wanted. JSMin itself is a really old and unsupported project anyway, so personally, I wouldn't bother.

I do all my asset compilation at build time now. There are lots of modern tools that help you do this, and are far superior to Combine in my opinion.

Catmacey commented 11 years ago

Hi Joe,

Yes like you we do minification/etc using other tools but this particular use was to compress some JS that's built automatically by CF from a large database query. In reality it's not that critical as it's fairly easy to control excess whitespace in the output anyway. It was more of a "because you can" kind of thing. But like you... I can't be bothered either... not if the next step is to go digging around in the Java source. :o)

Thanks again for the help.

Regards

Matt Casey

On 11 June 2013 15:07, Joe Roberts notifications@github.com wrote:

Hi Matt.

The JSmin source is available here https://github.com/zefer/Combine/blob/master/com/magnoliabox/jsmin/JSMin.java- so you could isolate it further by picking the classes it uses, if you wanted. JSMin itself is a really old and unsupported project anyway, so personally, I wouldn't bother.

I do all my asset compilation at build time now. There are lots of modern tools that help you do this, and are far superior to Combine in my opinion.

— Reply to this email directly or view it on GitHubhttps://github.com/zefer/Combine/issues/4#issuecomment-19260149 .

zefer / Combine

Error encoding JS files with UTF-8 encoding #4