New UTF-8 encoding issues

GoogleCodeExporter commented 9 years ago

I'm working on a project where we use wro4j, and recently we tried to upgrade 
from 1.3.6 to 1.3.8, but ran into some character encoding problems. Our project 
uses the YUI3 JS library, and one of the files is being served with improper 
encoding, which results in a parse error in Javascript when the page loads. The 
file in question is here:

https://github.com/yui/yui3/blob/3.3.0/build/text/text-data-wordbreak.js

On line 12, the value for the variable "aletter". When we used 1.3.6, this 
would encode properly and appear something like this:

aletter : 
'[A-Za-zªµºÀ-ÖØ-öø-ˁˆ-ˑˠ-ˤˬˮͰ-ʹͶͷͺ-ͽΆΈ-ΊΌΎ-ΡΣ-ϵϷ-�
�Ҋ-ԧԱ-Ֆՙա-ևא-תװ-׳ؠ-يٮٯٱ-ۓەۥۦۮۯۺ-ۼۿܐܒ-ܯݍ-ޥޱߊ-ߪߴ
ߵߺࠀ-ࠕࠚࠤࠨࡀ-ࡘऄ-हऽॐक़-ॡॱ-ॷॹ-ॿঅ-ঌএঐও
-নপ-রলশ-হঽৎড়ঢ়য়-ৡৰৱਅ-ਊਏਐਓ-ਨਪ-�
��ਲਲ਼ਵਸ਼ਸਹਖ਼-ੜਫ਼ੲ-ੴઅ-ઍએ-ઑઓ-નપ-રલ
ળવ-હઽૐૠૡଅ-ଌଏଐଓ-ନପ-ରଲଳଵ-ହଽଡ଼ଢ଼ୟ
-ୡୱஃஅ-ஊஎ-ஐஒ-கஙசஜஞடணதந-பம-ஹௐఅ-ఌఎ-
ఐఒ-నప-ళవ-హఽౘౙౠౡಅ-ಌಎ-ಐಒ-ನಪ-ಳವ-ಹಽೞ
ೠೡೱೲഅ-ഌഎ-ഐഒ-ഺഽൎൠൡൺ-ൿඅ-ඖක-නඳ-රලව-
ෆༀཀ-ཇཉ-ཬྈ-ྌႠ-Ⴥა-ჺჼᄀ-ቈቊ-ቍቐ-ቖቘቚ-ቝበ-�
��ኊ-ኍነ-ኰኲ-ኵኸ-ኾዀዂ-ዅወ-ዖዘ-ጐጒ-ጕጘ-ፚᎀ-ᎏᎠ-�
��ᐁ-ᙬᙯ-ᙿᚁ-ᚚᚠ-ᛪᛮ-ᛰᜀ-ᜌᜎ-ᜑᜠ-ᜱᝀ-ᝑᝠ-ᝬᝮ-ᝰ�
��-ᡷᢀ-ᢨᢪᢰ-ᣵᤀ-ᤜᨀ-ᨖᬅ-ᬳᭅ-ᭋᮃ-ᮠᮮᮯᯀ-ᯥᰀ-ᰣ��
�-ᱏᱚ-ᱽᳩ-ᳬᳮ-ᳱᴀ-ᶿḀ-ἕἘ-Ἕἠ-ὅὈ-Ὅὐ-ὗὙὛὝὟ
-ώᾀ-ᾴᾶ-ᾼιῂ-ῄῆ-ῌῐ-ΐῖ-Ίῠ-Ῥῲ-ῴῶ-ῼⁱⁿₐ-ₜ�
��ℇℊ-ℓℕℙ-ℝℤΩℨK-ℭℯ-ℹℼ-ℿⅅ-ⅉⅎⅠ-ↈⒶ-ⓩⰀ-Ⱞ
ⰰ-ⱞⱠ-ⳤⳫ-ⳮⴀ-ⴥⴰ-ⵥⵯⶀ-ⶖⶠ-ⶦⶨ-ⶮⶰ-ⶶⶸ-ⶾⷀ-ⷆ
ⷈ-ⷎⷐ-ⷖⷘ-ⷞⸯ々〻〼ㄅ-ㄭㄱ-ㆎㆠ-ㆺꀀ-ꒌꓐ-ꓽꔀ-ꘌꘐ-�
��ꘪꘫꙀ-ꙮꙿ-ꚗꚠ-ꛯꜗ-ꜟꜢ-ꞈꞋ-ꞎꞐꞑꞠ-ꞩꟺ-ꠁꠃ-ꠅꠇ
-ꠊꠌ-ꠢꡀ-ꡳꢂ-ꢳꣲ-ꣷꣻꤊ-ꤥꤰ-ꥆꥠ-ꥼꦄ-ꦲꧏꨀ-ꨨꩀ-ꩂ
ꩄ-ꩋꬁ-ꬆꬉ-ꬎꬑ-ꬖꬠ-ꬦꬨ-ꬮꯀ-ꯢ가-힣ힰ-ퟆퟋ-ퟻﬀ-ﬆﬓ
-ﬗיִײַ-ﬨשׁ-זּטּ-לּמּנּסּףּפּצּ-ﮱﯓ-ﴽﵐ-ﶏﶒ-
ﷇﷰ-ﷻﹰ-ﹴﹶ-ﻼＡ-Ｚａ-ｚﾠ-ﾾￂ-ￇￊ-ￏￒ-ￗￚ-ￜ]'

After upgrading to 1.3.8, it appears like this: 

aletter : 
'[A-Za-zªµºÀ-ÖØ-öø-?ˆ-??-????-????-???-???-??-??-??-??-???-??-??-??-???
?-???????-????-??-???-?????-?????-??-????-??-??-??-????-??-???-??????-????-????-
??-????????-???-??-??-??-??-????-??????-????-??-????-?????-????-??-??-?????????-
??-???-??-??-??-??-???????-??-??-??-??-????????-??-??-??????-??-??-??-???-???-??
-??-??-??-???-??-??-???-??-??-??-??-??-???-??-??-??-??-??-??-??-??-??-??-??-??-?
?-??-??-??-??-??-??-???-??-??-??-??-??-????-??-??-??-??-??-??-??-??-??-??-??-???
??-??-??-???-??-??-??-??-??-??-????-????-???-?????-??-??-??-???-??-??-??-??-??-?
?-??-???-??-??-??-??-??-??-??-??-??????-??-??-??-??-??-??-????-??-??-??-??-??-??
??-??-??-??-??-??-??-??-???-??-??-??-???-??-??-??-??-??-??-??-??-??-??-??-??-??-
???-??-??-???????-??-??-??-??-??-??-??-??-??-??-??-??-??-?]'

This issue only affects our devs running wro4j locally on Mac OS 10.6.8. They 
previously ran wro4j 1.3.6 without issue. We have a developer running Ubuntu 
10.04 LTS, and he sees the content being served correctly after upgrading to 
1.3.8.

Original issue reported on code.google.com by ghi...@gmail.com on 22 Jul 2011 at 8:58

GoogleCodeExporter commented 9 years ago

Do you use the maven plugin or the filter?

Original comment by alex.obj...@gmail.com on 23 Jul 2011 at 9:20

GoogleCodeExporter commented 9 years ago

Original comment by alex.obj...@gmail.com on 23 Jul 2011 at 9:21

Changed state: Accepted

GoogleCodeExporter commented 9 years ago

Also, could you provide more details about your configuration, processors used, 
etc. Thanks!

Original comment by alex.obj...@gmail.com on 23 Jul 2011 at 9:39

GoogleCodeExporter commented 9 years ago

[deleted comment]

GoogleCodeExporter commented 9 years ago

Not able to reproduce it on wro4j-examples (branch called 'dsl'). Waiting for 
more details regarding the processors you are using and if you are running 
wro4j as a maven plugin or a filter.

Original comment by alex.obj...@gmail.com on 23 Jul 2011 at 5:13

GoogleCodeExporter commented 9 years ago

We get the error running as the Maven plugin with 
<wroManagerFactory>ro.isdc.wro.extensions.manager.standalone.YUIStandaloneManage
rFactory</wroManagerFactory>

and as a filter with these initParams
    <init-param>
      <param-name>uriLocators</param-name>
      <param-value>servletContext,classpath,url</param-value>
    </init-param>
    <init-param>
      <param-name>preProcessors</param-name>
      <param-value>bomStripper,cssImport,cssUrlRewriting,lineNumbers,semicolonAppender,yuiJsMinAdvanced,yuiCssMin</param-value>
    </init-param>
    <init-param>
      <param-name>postProcessors</param-name>
      <param-value>cssVariables</param-value>
    </init-param>

(the lineNumbers is a custom processor that add PHP Minify type original line 
#s (be happy to contribute))

Processor is Intel Mac

Original comment by gcon...@gmail.com on 25 Jul 2011 at 7:12

GoogleCodeExporter commented 9 years ago

Could you try to remove the lineNumbers processors to see if it breaks the 
encoding? I'm trying to reproduce it locally and the result is as expected (no 
encoding problem) for the processors you've supplied. 

Still, I'm able to reproduce the problem for maven plugin.

Original comment by alex.obj...@gmail.com on 25 Jul 2011 at 9:26

GoogleCodeExporter commented 9 years ago

Original comment by alex.obj...@gmail.com on 26 Jul 2011 at 8:35

GoogleCodeExporter commented 9 years ago

Original comment by alex.obj...@gmail.com on 30 Jul 2011 at 1:26

Changed state: Duplicate

GoogleCodeExporter commented 9 years ago

Apparently the BomStripperPreProcessor messed the encoding. As a result of 
removing this preProcessor and removing BOM characters by default (when reading 
resource content), the encoding issue disappears.

Original comment by alex.obj...@gmail.com on 30 Jul 2011 at 1:32

GoogleCodeExporter commented 9 years ago

We just upgraded to 1.4.2 and this issue seems to have returned

Original comment by ghi...@gmail.com on 22 Dec 2011 at 6:38

GoogleCodeExporter commented 9 years ago

Posted a fix here

https://github.com/alexo/wro4j/pull/17

Original comment by gcon...@gmail.com on 23 Dec 2011 at 12:13

GoogleCodeExporter commented 9 years ago

Thanks for the patch, it was included in 1.4.3 release.

Original comment by alex.obj...@gmail.com on 24 Dec 2011 at 6:11

Changed state: Fixed
Added labels: Milestone-Release-1.4.3

Letractively / wro4j

New UTF-8 encoding issues #253