Encoding issues on Windows

[lucaswerkmeister]

Reported by @quintesse (in the ceylon.formatter repo):

$ ceylon run ceylon.formatter/1.3.2-SNAPSHOT source --to source_formatted
java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
        at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
        at java.io.BufferedWriter.close(BufferedWriter.java:265)
        at ceylon.file.internal.ConcreteFile$Overwriter.close(ConcreteFile.ceylon:183)
        at ceylon.file.Writer$impl.destroy(Writer.ceylon:26)
        at ceylon.file.File$impl$Overwriter.destroy(File.ceylon)
        at ceylon.formatter.format_.format(format.ceylon:77)
        at ceylon.formatter.format_.format(format.ceylon:68)
        at ceylon.formatter.run_.run(run.ceylon:296)
        at ceylon.formatter.run_.run(run.ceylon:277)
        at ceylon.formatter.run_.main(run.ceylon)

On Windows, the default encoding is not UTF-8. ANTLRFileStream has a two-parameter constructor that takes an encoding, which we should use. The only question is… which charset? Can we access the Ceylon config for this?

[quintesse]

If you're running as a CeylonTool you should just be able to get a reference to CeylonConfig or better yet to DeafultToolOptions.getDefaultEncoding() (or something similar, I'm typing from memory here)

[lucaswerkmeister]

I already have some config-related code lying around somewhere, for reading profiles. I’ll have to look at that. (I guess I should check that ceylon format and ceylon run ceylon.formatter handle configs in the same way…)

[lucaswerkmeister]

@quintesse can you test the fix-133 branch? (ant install, then ceylon format source/ and ceylon run ceylon.formatter/1.3.2-SNAPSHOT source/, should not result in encoding errors.)

[lucaswerkmeister]

Okay, better version pushed now using the tools API. (Which, I hope, will continue to work when run as ceylon run in the future.)

[quintesse]

@lucaswerkmeister I can build the entire distribution (+sdk +eclipse) just fine using this branch but running ant install results in many dozens of unmappable character errors like:

[ceylon-compile] C:\Projects\ceylon\ceylon.formatter\source\ceylon\formatter\FormattingWriter.ceylon:31: error: Ceylon backend error: unmappable character for encoding UTF-8
[ceylon-compile] "The maximum value that is safe to use as [[FormattingWriter.writeToken]]?s `space[Before|After]` argument.

The same when running ceylon format source/ just after having built the distribution.

[lucaswerkmeister]

In ceylon-compile‽ That doesn’t make sense, there’s an encoding specified in .ceylon/config…

[lucaswerkmeister]

Wait, no, the error message says it’s using UTF-8! What?

[quintesse]

Ok, GIt checks it out as ASCII, not UTF8

[quintesse]

Ok, I've got the locale set correctly now so ant install works, but ceylon format source/ still results in:

java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
        at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
        at java.io.BufferedWriter.close(BufferedWriter.java:265)
        at ceylon.file.internal.ConcreteFile$Overwriter.close(ConcreteFile.ceylo                                            n:183)
        at ceylon.file.Writer$impl.destroy(Writer.ceylon:26)
        at ceylon.file.File$impl$Overwriter.destroy(File.ceylon)
        at ceylon.formatter.format_.format(format.ceylon:77)
        at ceylon.formatter.format_.format(format.ceylon:68)
        at ceylon.formatter.run_.run(run.ceylon:293)
        at ceylon.formatter.CeylonFormatTool.run(CeylonFormatTool.ceylon:69)

.
.
.
ceylon format: Input length = 1
java.nio.charset.UnmappableCharacterException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:282)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
        at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
        at java.io.BufferedWriter.write(BufferedWriter.java:230)
        at java.io.Writer.write(Writer.java:157)
        at ceylon.file.internal.ConcreteFile$Overwriter.write(ConcreteFile.ceylo                                            n:187)
        at ceylon.formatter.recoveryOnError_$1.$call$(run.ceylon:45)
        at ceylon.formatter.run_.run(run.ceylon:295)
        at ceylon.formatter.CeylonFormatTool.run(CeylonFormatTool.ceylon:69)
.
.
.
        Suppressed: java.nio.charset.UnmappableCharacterException: Input length                                             = 1
                at java.nio.charset.CoderResult.throwException(CoderResult.java:                                            282)
                at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:285)
                at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
                at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
                at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129)
                at java.io.BufferedWriter.close(BufferedWriter.java:265)
                at ceylon.file.internal.ConcreteFile$Overwriter.close(ConcreteFi                                            le.ceylon:183)
                at ceylon.file.Writer$impl.destroy(Writer.ceylon:26)
                at ceylon.file.File$impl$Overwriter.destroy(File.ceylon)
                at ceylon.formatter.recoveryOnError_$1.$call$(run.ceylon:44)
                ... 19 more

[lucaswerkmeister]

Right, I need to set the encoding when writing, too. Try again, please?

[quintesse]

@lucaswerkmeister yup! works fine now :)

[lucaswerkmeister]

Great, thanks!