`hGetContents: invalid argument (invalid byte sequence)` on Japanese Windows

[igrep]

My environment

OS: Windows 10 Pro ver. 1709 build 16299.431 Editor: NVIM v0.2.2 LSP Client: https://github.com/autozimu/LanguageClient-neovim ver. 0.1.93

HIE (extracted from the log file):

hie-wrapper(hie-wrapper) Version 0.2.0.0, Git revision 2a8bad55422cb436b18660dd9238c315b1d9bf82 (1545 commits) x86_64 ghc-8.2.2
HIE(hie-8.2.2.exe) Version 0.2.0.0, Git revision 2a8bad55422cb436b18660dd9238c315b1d9bf82 (1545 commits) x86_64 ghc-8.2.2

Problem

An error is reported when opening a file in a project where some files contains Japanese characters. From the debug log:

2018-07-04 10:30:51.3777965 [ThreadId 5] - <--2--{"jsonrpc":"2.0","params":{"type":1,"message":"Got error while processing diagnostics: C:\\Users\\yuji-yamamoto\\AppData\\Local\\Temp\\ghc-mod32077\\Experiment32076-0.hs: hGetContents: invalid argument (invalid byte sequence)"},"method":"window/showMessage"}

Suggestion and question

I know the error invalid argument (invalid byte sequence) is a very common error on Japanese Windows related to character encoding. I've once fixed very similar one in haddock: https://github.com/haskell/haddock/pull/566. So the solution is perhaps to add hSetEncoding handle utf8 before reading a source file.

Then, where should I fix? According to the next line of the error in the debug log, hlint is successfully executed:

2018-07-04 10:30:51.3778748 [ThreadId 5] - <--2--{"jsonrpc":"2.0","params":{"uri":"file:///C%3A/s/pj/shioyaki/vendor/haskell-relational-record/relational-query/src/Database/Relational/Experiment.hs","diagnostics":[{"severity":4,"range":{"start":{"line":49,"character":17},"end":{"line":49,"character":83}},"code":"Redundant bracket","source":"hlint","message":"Redundant bracket\nFound:\n  unPlaceholders :: (IxStateT (WriterT PlaceHolderValues m) p1 p2 a)\nWhy not:\n  unPlaceholders :: IxStateT (WriterT PlaceHolderValues m) p1 p2 a\n"},{"severity":4,"range":{"start":{"line":187,"character":17},"end":{"line":187,"character":24}},"code":"Use const","source":"hlint","message":"Use const\nFound:\n  \\ _ -> k\nWhy not:\n  const k\n"}]},"method":"textDocument/publishDiagnostics"}

So the error is caused not by hlint, but ghc-mod, right?

[lukel97]

Do you an example of a file that can reproduce this?

[igrep]

Okay, here's the file HIE fails to read with the error: https://github.com/igrep/haskell-relational-record/blob/d3cda009f1cbbcf1e542bcce51cfaa9a6b47bff5/relational-query/src/Database/Relational/Experiment.hs.

[lukel97]

I can't seem to recreate it with just that file, can you try capturing the session with the hie-8.2.2 --lsp -c session.json? Also do you have any stack.yaml/other files/dependencies needed to build that example?

[igrep]

Oh, sorry, I should have made a smaller example. Perhaps putting a comment below on any .hs file would cause the error:

-- 書き換えるのは難しい？

But if you can't, follow these steps:

chcp 932 # Switch the locale into Japanese.
stack new some-project simple
cd some-project
echo '-- 書き換えるのは難しい？' >> src/Main.hs
nvim src/Main.hs

[igrep]

And here's the session info getting the error. NOTE: I modified the source a little to surely reproduce the error.

{"tag":"FromClient","contents":["2018-07-05T00:51:44.5043342Z",{"tag":"ReqInitialize","contents":{"jsonrpc":"2.0","params":{"rootUri":"file:///C:/Users/yuji-yamamoto/Downloads/some-project","processId":6436,"rootPath":"C:\\Users\\yuji-yamamoto\\Downloads\\some-project","capabilities":{"workspace":{"didChangeWatchedFiles":{"dynamicRegistration":true},"applyEdit":true},"textDocument":{"completion":{"completionItem":{"snippetSupport":true}}}},"trace":"off"},"method":"initialize","id":9}}]}
{"tag":"FromServer","contents":["2018-07-05T00:51:44.5395614Z",{"tag":"RspInitialize","contents":{"result":{"capabilities":{"textDocumentSync":{"openClose":true,"change":2,"willSave":false,"willSaveWaitUntil":false,"save":{"includeText":false}},"documentRangeFormattingProvider":true,"documentHighlightProvider":true,"executeCommandProvider":{"commands":["12200:applyrefact:applyOne","12200:hare:demote"]},"renameProvider":true,"definitionProvider":true,"hoverProvider":true,"codeActionProvider":true,"completionProvider":{"triggerCharacters":["."],"resolveProvider":true},"documentSymbolProvider":true,"documentFormattingProvider":true,"referencesProvider":true}},"jsonrpc":"2.0","id":9}}]}
{"tag":"FromClient","contents":["2018-07-05T00:51:44.5469047Z",{"tag":"NotInitialized","contents":{"jsonrpc":"2.0","params":{},"method":"initialized"}}]}
{"tag":"FromClient","contents":["2018-07-05T00:51:44.548928Z",{"tag":"NotDidOpenTextDocument","contents":{"jsonrpc":"2.0","params":{"textDocument":{"languageId":"haskell","text":"module Main where\n\nmain :: IO ()\nmain = do\n  putStrLn \"hello world\"\n-- 書き換えるのは難しい？\n-- 新しい日本語の文章を考えよう\n","uri":"file:///C:/Users/yuji-yamamoto/Downloads/some-project/src/Main.hs","version":0}},"method":"textDocument/didOpen"}}]}
{"tag":"FromServer","contents":["2018-07-05T00:51:44.5559955Z",{"tag":"NotLogMessage","contents":{"jsonrpc":"2.0","params":{"type":4,"message":"Using hie version: Version 0.2.0.0, Git revision 2a8bad55422cb436b18660dd9238c315b1d9bf82 (1545 commits) x86_64 ghc-8.2.2"},"method":"window/logMessage"}}]}
{"tag":"FromServer","contents":["2018-07-05T00:51:45.0126648Z",{"tag":"NotLogMessage","contents":{"jsonrpc":"2.0","params":{"type":4,"message":"Using hoogle db at: C:\\Users\\yuji-yamamoto\\AppData\\Roaming\\hoogle\\default-haskell-5.0.17.hoo"},"method":"window/logMessage"}}]}
{"tag":"FromServer","contents":["2018-07-05T00:51:45.8083646Z",{"tag":"NotPublishDiagnostics","contents":{"jsonrpc":"2.0","params":{"uri":"file:///C%3A/Users/yuji-yamamoto/Downloads/some-project/src/Main.hs","diagnostics":[{"severity":3,"range":{"start":{"line":3,"character":7},"end":{"line":4,"character":24}},"code":"Redundant do","source":"hlint","message":"Redundant do\nFound:\n  do putStrLn \"hello world\"\nWhy not:\n  putStrLn \"hello world\"\n"}]},"method":"textDocument/publishDiagnostics"}}]}
{"tag":"FromServer","contents":["2018-07-05T00:51:46.2290997Z",{"tag":"NotShowMessage","contents":{"jsonrpc":"2.0","params":{"type":1,"message":"Got error while processing diagnostics: C:\\Users\\yuji-yamamoto\\AppData\\Local\\Temp\\ghc-mod12201\\Main12200-0.hs: hGetContents: invalid argument (invalid byte sequence)"},"method":"window/showMessage"}}]}
{"tag":"FromServer","contents":["2018-07-05T00:51:46.2301008Z",{"tag":"NotPublishDiagnostics","contents":{"jsonrpc":"2.0","params":{"uri":"file:///C%3A/Users/yuji-yamamoto/Downloads/some-project/src/Main.hs","diagnostics":[{"severity":3,"range":{"start":{"line":3,"character":7},"end":{"line":4,"character":24}},"code":"Redundant do","source":"hlint","message":"Redundant do\nFound:\n  do putStrLn \"hello world\"\nWhy not:\n  putStrLn \"hello world\"\n"}]},"method":"textDocument/publishDiagnostics"}}]}
{"tag":"FromClient","contents":["2018-07-05T00:51:51.2780931Z",{"tag":"NotExit","contents":{"jsonrpc":"2.0","params":null,"method":"exit"}}]}

[igrep]

When reproducing, I found the error does not always happen when opening files containing the problematic characters. It seems that creating a new project with the same source file but with a different name can reproduce. Is something cached?

[lukel97]

I just realised this is on windows, I'm on macOS so I will have to setup hie on a windows partition first and then get back to you!

[MaskRay]

Is this an encoding issue?

[igrep]

Yes, I guess HIE (precisely, ghc-mod in HIE) is assumes the file is a CP932 (Japanese Windows' default character encoding) file. But actually the file is a UTF-8 file.

[alanz]

TODO: apply ghc-mod fix.