encoding issues dependent on environment settings

[mrbrdo]

For reference: https://github.com/rails/rails/issues/12720

When minifying assets with accented characters (e.g. russian in Sugar.JS), when environment settings are in a certain way (see below), uglifier will replace those characters with question marks. Only tested through rails assets:precompile.

Important: I am just assuming it's the locale settings, I have no way of knowing. It is something that started being forwarded in net-ssh as of this change: https://github.com/net-ssh/net-ssh/blob/master/CHANGES.txt#L6

Failing machine locale:

LANG="sl_SI.UTF-8"
LC_COLLATE="sl_SI.UTF-8"
LC_CTYPE="sl_SI.UTF-8"
LC_MESSAGES="sl_SI.UTF-8"
LC_MONETARY="sl_SI.UTF-8"
LC_NUMERIC="sl_SI.UTF-8"
LC_TIME="sl_SI.UTF-8"
LC_ALL=

Another failing machine locale:

LANG=
LC_COLLATE="C"
LC_CTYPE="UTF-8"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

OK machine locale:

LANG=
LC_COLLATE="C"
LC_CTYPE="C"
LC_MESSAGES="C"
LC_MONETARY="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_ALL=

The only thing I can deduce from this is that both failing machines have some variation of UTF-8 for LC_CTYPE.

Uglifier 2.3.0.

PS: Neither yui-compressor or closure-compiler experience this problem, on identical setup.

[brendandc]

With the recent resolution of https://github.com/lautis/uglifier/issues/47, which was causing encoding to fail with UTF-8 chars, I just ran into the same problem with utf8 characters getting replaced by question marks in the minified javascript.

Uglifier 2.3.1 Rails 4.0.1 also using rake assets:precompile Doing the precompile locally on my Windows 7 machine, did not confirm on anything else.

[mrbrdo]

Well I can confirm it is 100% a locale-related issue for me. For example, I tried setting up the locale that was being forwarded through SSH to my server, and after that the problems went away. But I think uglifier should not depend on these settings.

I must note that yui-compressor does actually exhibit a similar problem. In the case I was trying at first, yui-compressor had no problems (while uglifier did), but I later came across the same problem with a different project, and yui-compressor couldn't handle it either.

[janusch]

Hey, i think i have a similar issue caused by uglifier. In production the app does not run on IE9 due to an invalid character, this is probably caused by uglifier. How did you work around this? Is it possible to limit the charset uglifier is using?

[lautis]

Uglifier has ascii_only output option, but it is set to true by default since 2.5.0. If you run into invalid output, it might be worth checking if UglifyJS produces similar output.