Error when using cpphs in some locale environments

[asr]

Some Agda users have reported an error when installing Agda in their locale environments.

A MWE (adapted from this example) is the following:

$ cat Test.hs
module Main where

main = putStrLn "∀"

$ LC_CTYPE=C cpphs Test.hs > /dev/null
cpphs: Test.hs: hGetContents: invalid argument (invalid byte sequence)

@nad wrote here:

I guess that cpphs uses the standard, locale-aware methods to read files. I think all of our source files use the UTF-8 character encoding, so the problem can perhaps be solved by setting LC_CTYPE to .UTF-8 before invoking cpphs, for some locale .UTF-8 that is installed. However, I would not be surprised if it is impossible to do this in a system-independent way. Perhaps it would be better to add a --utf8 flag to cpphs.

Blocking https://github.com/agda/agda/issues/2112.

[malcolmwallace]

I can't seem to reproduce the issue with the given steps. cpphs uses the standard Haskell/ghc System.IO.openFile, which I think trusts the underlying filesystem's metadata about the file's encoding? Certainly, setting LC_CTYPE does not seem to change its behaviour.

$ LC_CTYPE=C ./cpphs Test.hs 
#line 1 "Test.hs"
module Main where

main = putStrLn "∀"

[asr]

Using the file command for determining the file type I got

$ file Test.hs
Test.hs: UTF-8 Unicode text

What do you get?

[malcolmwallace]

The same.

[asr]

It seems you have no the C locale installed. Which is the output of running

$ locale -a

?

[malcolmwallace]

$ locale -a af_ZA af_ZA.ISO8859-1 af_ZA.ISO8859-15 af_ZA.UTF-8 am_ET am_ET.UTF-8 be_BY be_BY.CP1131 be_BY.CP1251 be_BY.ISO8859-5 be_BY.UTF-8 bg_BG bg_BG.CP1251 bg_BG.UTF-8 ca_ES ca_ES.ISO8859-1 ca_ES.ISO8859-15 ca_ES.UTF-8 cs_CZ cs_CZ.ISO8859-2 cs_CZ.UTF-8 da_DK da_DK.ISO8859-1 da_DK.ISO8859-15 da_DK.UTF-8 de_AT de_AT.ISO8859-1 de_AT.ISO8859-15 de_AT.UTF-8 de_CH de_CH.ISO8859-1 de_CH.ISO8859-15 de_CH.UTF-8 de_DE de_DE.ISO8859-1 de_DE.ISO8859-15 de_DE.UTF-8 el_GR el_GR.ISO8859-7 el_GR.UTF-8 en_AU en_AU.ISO8859-1 en_AU.ISO8859-15 en_AU.US-ASCII en_AU.UTF-8 en_CA en_CA.ISO8859-1 en_CA.ISO8859-15 en_CA.US-ASCII en_CA.UTF-8 en_GB en_GB.ISO8859-1 en_GB.ISO8859-15 en_GB.US-ASCII en_GB.UTF-8 en_IE en_IE.UTF-8 en_NZ en_NZ.ISO8859-1 en_NZ.ISO8859-15 en_NZ.US-ASCII en_NZ.UTF-8 en_US en_US.ISO8859-1 en_US.ISO8859-15 en_US.US-ASCII en_US.UTF-8 es_ES es_ES.ISO8859-1 es_ES.ISO8859-15 es_ES.UTF-8 et_EE et_EE.ISO8859-15 et_EE.UTF-8 eu_ES eu_ES.ISO8859-1 eu_ES.ISO8859-15 eu_ES.UTF-8 fi_FI fi_FI.ISO8859-1 fi_FI.ISO8859-15 fi_FI.UTF-8 fr_BE fr_BE.ISO8859-1 fr_BE.ISO8859-15 fr_BE.UTF-8 fr_CA fr_CA.ISO8859-1 fr_CA.ISO8859-15 fr_CA.UTF-8 fr_CH fr_CH.ISO8859-1 fr_CH.ISO8859-15 fr_CH.UTF-8 fr_FR fr_FR.ISO8859-1 fr_FR.ISO8859-15 fr_FR.UTF-8 he_IL he_IL.UTF-8 hi_IN.ISCII-DEV hr_HR hr_HR.ISO8859-2 hr_HR.UTF-8 hu_HU hu_HU.ISO8859-2 hu_HU.UTF-8 hy_AM hy_AM.ARMSCII-8 hy_AM.UTF-8 is_IS is_IS.ISO8859-1 is_IS.ISO8859-15 is_IS.UTF-8 it_CH it_CH.ISO8859-1 it_CH.ISO8859-15 it_CH.UTF-8 it_IT it_IT.ISO8859-1 it_IT.ISO8859-15 it_IT.UTF-8 ja_JP ja_JP.SJIS ja_JP.UTF-8 ja_JP.eucJP kk_KZ kk_KZ.PT154 kk_KZ.UTF-8 ko_KR ko_KR.CP949 ko_KR.UTF-8 ko_KR.eucKR lt_LT lt_LT.ISO8859-13 lt_LT.ISO8859-4 lt_LT.UTF-8 nl_BE nl_BE.ISO8859-1 nl_BE.ISO8859-15 nl_BE.UTF-8 nl_NL nl_NL.ISO8859-1 nl_NL.ISO8859-15 nl_NL.UTF-8 no_NO no_NO.ISO8859-1 no_NO.ISO8859-15 no_NO.UTF-8 pl_PL pl_PL.ISO8859-2 pl_PL.UTF-8 pt_BR pt_BR.ISO8859-1 pt_BR.UTF-8 pt_PT pt_PT.ISO8859-1 pt_PT.ISO8859-15 pt_PT.UTF-8 ro_RO ro_RO.ISO8859-2 ro_RO.UTF-8 ru_RU ru_RU.CP1251 ru_RU.CP866 ru_RU.ISO8859-5 ru_RU.KOI8-R ru_RU.UTF-8 sk_SK sk_SK.ISO8859-2 sk_SK.UTF-8 sl_SI sl_SI.ISO8859-2 sl_SI.UTF-8 sr_YU sr_YU.ISO8859-2 sr_YU.ISO8859-5 sr_YU.UTF-8 sv_SE sv_SE.ISO8859-1 sv_SE.ISO8859-15 sv_SE.UTF-8 tr_TR tr_TR.ISO8859-9 tr_TR.UTF-8 uk_UA uk_UA.ISO8859-5 uk_UA.KOI8-U uk_UA.UTF-8 zh_CN zh_CN.GB18030 zh_CN.GB2312 zh_CN.GBK zh_CN.UTF-8 zh_CN.eucCN zh_HK zh_HK.Big5HKSCS zh_HK.UTF-8 zh_TW zh_TW.Big5 zh_TW.UTF-8 C POSIX

[malcolmwallace]

I don't know whether the version of ghc might be relevant, but in case it is, I'm compiling cpphs with ghc-7.6.1

[asr]

You have the C locale installed. I could reproduce the issue compiling cpphs with GHC 7.6.3. What shell are you using? I'm using

$ echo $SHELL
/bin/bash

[nad]

cpphs uses the standard Haskell/ghc System.IO.openFile, which I think trusts the underlying filesystem's metadata about the file's encoding?

I think recent versions of GHC by default use the locale (or code page) to decide what encoding to use.

[nad]

A simple (system-dependent) test:

$ echo -e '\u2200' > test
$ cat test
∀
$ file test
test: UTF-8 Unicode text
$ ghc -e 'putStr =<< readFile "test"'
∀
$ LC_CTYPE=C ghc -e 'putStr =<< readFile "test"'
<interactive>: test: hGetContents: invalid argument (invalid byte sequence)

[nad]

Certainly, setting LC_CTYPE does not seem to change its behaviour.

Perhaps you've set LC_ALL, which overrides LC_CTYPE.

[malcolmwallace]

$ ghc --version The Glorious Glasgow Haskell Compilation System, version 7.8.4 $ cat test ∀ $ file test test: UTF-8 Unicode text $ ghc -e 'putStr =<< readFile "test"' ∀ $ LC_CTYPE=C ghc -e 'putStr =<< readFile "test"' ∀ $ LC_ALL=C ghc -e 'putStr =<< readFile "test"' ∀

[malcolmwallace]

I think I can close this issue, since it appears that neither cpphs nor ghc is at fault.

[asr]

Which operating system and shell are you using?

[asr]

Could you reproduce the issue running

$ export LC_ALL=C
$ cpphs test

?

[malcolmwallace]

ghc-7.6.1 on MacOSX 10.7.5, with bash. ghc-7.8.3 on Windows 7 Professional SP1, with bash.

[malcolmwallace]

Cannot reproduce the issue, even with LC_ALL=C.

[asr]

Did you mean export LC_ALL=C?

[asr]

Which is the output of

$ locale
$ LC_ALL=C locale

?

[malcolmwallace]

$ locale # MacOSX LANG="en_GB.UTF-8" LC_COLLATE="en_GB.UTF-8" LC_CTYPE="en_GB.UTF-8" LC_MESSAGES="en_GB.UTF-8" LC_MONETARY="en_GB.UTF-8" LC_NUMERIC="en_GB.UTF-8" LC_TIME="en_GB.UTF-8" LC_ALL= $ LC_ALL=C locale LANG="en_GB.UTF-8" LC_COLLATE="C" LC_CTYPE="C" LC_MESSAGES="C" LC_MONETARY="C" LC_NUMERIC="C" LC_TIME="C" LC_ALL="C"

The result is similar on Windows 7, except that the default is en_US.UTF-8 rather than en_GB.UTF-8.

[nad]

ghc-7.6.1 on MacOSX 10.7.5, with bash. ghc-7.8.3 on Windows 7 Professional SP1, with bash.

I just discussed this issue with a Mac user, and it seems as if the System.IO functions by default always use UTF-8 under MacOS, while the locale is ignored.

Under Windows I guess that one can use chcp to trigger the problem. Perhaps chcp 1252 would work.

GHC has used UTF-8 as the character encoding for source files since version 6.6 (which was released in 2006), so perhaps cpphs could also use this as the default. Note, however, that the GHC documentation states that "invalid UTF-8 sequences [are] ignored in comments, so it is possible to use other encodings such as Latin-1, as long as the non-comment source code is ASCII only".

I've attached a patch that switches to UTF-8 everywhere (?) in cpphs, with two caveats:

• The command-line arguments are treated as before.
• The encoding of stderr is only changed in the top-level module. If cpphs is intended to be used as a library, and error messages can contain non-ASCII characters, then the encoding of stderr should perhaps be changed in the applicable library modules.

I've used the base library's support for roundtripping to handle illegal characters. Feel free to base any changes on this patch.

[asr]

FYI, I reported here the different behaviour in Linux and Mac OS.

[malcolmwallace]

Thanks for the patch Nils. I rolled something slightly different, to ensure that e.g. #included files also get the UTF8 encoding. I was not previously aware of the roundtripping style of TextEncoding, so that was a useful addition for me.

[asr]

Thanks for fixing the issue (tested on Agda). Could you release a new version, please.

[malcolmwallace]

cpphs-1.20.2 released.