Encoding issue on Windows

[rpiaggio]

https://gist.github.com/rpiaggio/1e08ae305a14253da97b

In the Gist above, with expected output (produced by http://www.sassmeister.com/) line 19 shows correct UTF-8 character.

When running sbt-sassify on Linux, this works correctly.

However, when running it on Windows, that line is output as .fa-linkedin:before{content:"?"} with an actual question mark (HEX 3F) in the file.

I'm not sure this is a sbt-sassify issue or an internal libsass issue.

Thank you! (and thank you for sbt-sassify)

[rpiaggio]

It seems to be a sbt-sassify issue.

Working with grunt-sass on Windows works fine.

[rpiaggio]

After a bit more investigation, this seems to be originated by the way JNA converts strings: https://jna.java.net/javadoc/overview-summary.html#strings.

From there: When converting Java unicode characters into an array of char, the default platform encoding is used, unless the system property jna.encoding is set to a valid encoding. This property may be set to "UTF8", for example, to ensure all native strings use that encoding.

This would explain why it works in Linux and not on Windows. However, I called sbt with -Djna.encoding=UTF8 and the problem persists.

There is a related question on StackOverflow: http://stackoverflow.com/questions/35871861/jna-result-character-encoding-result-in-utf8, which was ultimately solved by calling another API method that returned wide chars, not available in the case of libsass. However, the user reports jna.encoding not having an effect on output strings.

[irundaia]

I was looking into this issue already. I can confirm that it seems to be an encoding issue. I think this issue is occurring when writing the output CSS to a file. (When debugging, I can see that the compiled string is consistent with the output from my OS X results.)

[rpiaggio]

Hi @irundaia!

Are you using OS X then? I don't know that OS X default encoding is, does the issue happen there? I can confirm that it doesn't happen on Linux.

I was following the JNA path... If JNA is the issue, then I think this could be solved by receiving the CSS output as a Pointer instead of a String in https://github.com/irundaia/sbt-sassify/blob/master/src/org/irundaia/sass/jna/SassLibrary.java#L1357, and then calling pointer.getString(0, "UTF-8") (https://github.com/java-native-access/jna/blob/master/src/com/sun/jna/Pointer.java#L693).

What do you think?

[rpiaggio]

Or, if it is an issue when saving the file, it's just a matter of specifying UTF-8 character encoding in https://github.com/irundaia/sbt-sassify/blob/master/src/org/irundaia/sass/SassCompiler.scala#L69.

(Calling this: https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#write(java.nio.file.Path,%20java.lang.Iterable,%20java.nio.charset.Charset,%20java.nio.file.OpenOption...))

[irundaia]

Yup I've tested it on both OS X and Linux.

Now, the issue does not lie with the interface between JNA and the scala/java code but in the way I wrote the output to the destination file. I converted the String to an Array[Byte] using the String.toBytes() function. Apparently, this messes up the character encoding.

If I instead use String.toBytes(StandardCharsets.UTF_8), it seems to be working properly. I want to run some extra tests on both OS X and Linux to make sure that this does not break their output encoding.

In any case! thanks a bunch for the report!

[rpiaggio]

Ah, great! Thank you very much for fixing it. Sorry if I overwhelmed you with "off the track" info.

[irundaia]

Yup, it's the latter issue. I've tested it on both Linux and OS X, it seems to work there as well. I'm a bit worried though, that this is OS (perhaps even FS) dependant. In any case, I'll release the fix as it is. I hope this solves your issues. If not, please let me know.

[rpiaggio]

Thank you! I confirmed that the issue is fixed for me.

[irundaia]

Great, thanks!