Retrie is broken on Windows

[bradrn]

Cross-posted from https://github.com/emacs-lsp/lsp-haskell/issues/128 on the recommendation of @michaelpj:

When I try to apply a code action such as ‘Unfold’ or ‘Fold’, I get the following message:

LSP :: Please open an issue in lsp-mode for implementing `11160:retrie:retrieCommand'.

(error "Not enough arguments for format string")

And the relevant part of the log from LSP:

[Trace - 06:55:01 PM] Received response 'workspace/executeCommand - (95)' in 0ms.
Result: {
  "code": -32602,
  "message": "error while parsing args for retrieCommand in plugin retrie: expected Bool, but encountered Null\narg = Object (fromList [(\"description\",String \"Unfold wSheet\"),(\"originatingFile\",String \"file:///c%3A/Users/bradn/Documents/Spreadsheet/supercell/src/Main.hs\"),(\"restrictToOriginatingFile\",Null),(\"rewrites\",Array [Object (fromList [(\"contents\",String \"Main.wSheet\"),(\"tag\",String \"Unfold\")])])])"
}

[jneira]

Hi, thanks for the bug report, could you post the module where you are trying to trigger retry (or the relevant pragmas for retrie)?

[pepeiborra]

Could you also share a repro using VSCode?

[bradrn]

@jneira It fails for every module I’ve tried.

(Also, it’s not just retrie either: all Wingman operations fail as well.)

@pepeiborra I don’t use VSCode regularly, but I’ll try to reproduce it.

[bradrn]

@pepeiborra VSCode also gives me an error when I attempt to use Retrie or Wingman:

[pepeiborra]

Thanks @bradrn but we need more details to reproduce. Project, steps, versions of hls, ghc, etc.

[bradrn]

Project: this seems to happen for any project, even just plain Stack simple template.

GHC version: again, this seems to be independent of GHC version from what I can tell.

HLS version:

> .\haskell-language-server-wrapper.exe --version
haskell-language-server version: 1.2.0.0 (GHC: 8.10.4) (PATH: C:\Users\bradn\Documents\Haskell\hls\haskell-language-server-wrapper.exe) (GIT hash: be2071e985cb417f984ab00a1aad76dee02d6d0b)

[pepeiborra]

Can you attach a screen recording too please?

[bradrn]

(EDIT: The uploaded video seems to have cut off all the relevant parts of my original recording. I’ll try to upload a better video.)

[bradrn]

@pepeiborra Here’s a better recording:

https://user-images.githubusercontent.com/17613703/127493429-1c04d86c-43de-4692-8fca-8e70cf97bcf1.mp4

As shown towards the end of the video most parts of HLS are working correctly; it is only retrie and parts of Wingman which are buggy.

[jneira]

hmm from the path of the call to hls in the cli it seems you are on windows, no?

[bradrn]

Correct, I am on Windows 10.

[jneira]

ok, there is a existing issue about retrie and windows, although the symptoms are not the same afaics #533

[bradrn]

Interesting… not sure if this is the same issue, but they certainly could be related. Do you know of any further tests I could try to diagnose this? (Though it’s getting late here and I don’t have time to do anything just right now.)

[jneira]

I just tried a retrie fold and i am continue getting the same error as #533. Maybe your way to start emacs makes xargs to be in scope? Your error would be triggered after avoiding that

[bradrn]

Sorry, I’m not sure what you mean… are you saying that I should try removing xargs from PATH, or I should add it to PATH?

[jneira]

no, no, I think maybe you already have xargs in scope and the error you are getting is the next one will try to reproduce yours

[jneira]

However wingman works correctly in my end

[isovector]

Logs for the wingman actions would be helpful as well. Thanks!

[bradrn]

@isovector Sorry, which logs do you want? Because as far as I can tell the logs are the same for Retrie and Wingman — both give similar parse errors.

[isovector]

@bradrn maybe just the whole log!

[jneira]

Mmm, one thing that share retrie and wingman is ghc-exactprint usage iirc. No evidence to back the intuition though

[bradrn]

Curiouser and curiouser. I tried to test the various code actions available to confirm what does and doesn’t work, ~and to my surprise I could not find a single Wingman command which failed~ (EDIT: found one, see next comment). ~I’m not sure why I reported that Wingman was failing, but whatever it was, it seems to be working perfectly now.~

On the other hand, Retrie is still not working, but it seems to give different errors depending on which command I use:

• ‘Unfold’ and ‘Fold’ both give the parse error mentioned above
• ‘Unfold in current file’ gives no error, but has no effect either
• ‘Fold in current file’ fails with ExitFailure 1.

@isovector I tried collecting logs while testing, but I doubt you’ll be interested in all 70000+ lines! I’m not even sure where I would share a file of that size.

[bradrn]

OK, now I’m thoroughly stumped. I decided to try Wingman in another project, and this time round it failed, with an absolutely bizarre error message:

LSP :: `workspace/executeCommand' with `1836:tactics:tacticsDestructLambdaCaseCommand' failed.

(error "C:\\sr\\ghc-8.10.4\\hyphenation-0.8.1-f8b6bb68cfad4b10e18172bc9e3eb5622f323a16\\share\\hyph-en-us.hyp.txt.gz: openBinaryFile: does not exist (No such file or directory)")

Any ideas?

[jneira]

Ugh, do you have the package installed in that location? D:\sr seems the dir stack sets as %STACK_ROOT% by default when installed in windows, but ghc-8.10.4\\hyphenation-0.8.1-f8b6bb68cfad4b10e18172bc9e3eb5622f323a16 is the typical path cabal uses to install packages with the v2-command. Maybe did you delete it?

[jneira]

@isovector I tried collecting logs while testing, but I doubt you’ll be interested in all 70000+ lines! I’m not even sure where I would share a file of that size.

I think all lines emitted by wingman would have the plugin name, maybe extracting them from the log while reproducing the error (they should not be so much) would help

[bradrn]

Ugh, do you have the package installed in that location?

The path C:\sr\ghc-8.10.4 doesn’t even exist on my machine. I’m not sure where it’s coming from, unless it’s hardcoded in the source (which it surely wouldn’t be‽).

I think all lines emitted by wingman would have the plugin name, maybe extracting them from the log while reproducing the error (they should not be so much) would help

Good idea, I’ll try this.

[jneira]

The path C:\sr\ghc-8.10.4 doesn’t even exist on my machine. I’m not sure where it’s coming from, unless it’s hardcoded in the source (which it surely wouldn’t be‽).

very interesting, those paths could be hardcoded while assembling the executable (although they are not directly in the source code as you suspect), how did you got the hls executable? did the vscode extension download for you?

[bradrn]

As mentioned already I’m using Emacs; the executables are simply downloaded from the GitHub releases page.

[jneira]

As mentioned already I’m using Emacs; the executables are simply downloaded from the GitHub releases page.

oh, right, vscode downloads the executable from the same location. To investigate. I will try to dust off my emacs installation in my windows 10 laptop if i have time.

[isovector]

You're running into https://github.com/haskell/haskell-language-server/issues/2015 on the Wingman side

[bradrn]

Thanks @isovector!

[Tarmean]

I've been meaning to look into retrie on windows. My current thinking is that this is an escaping issues when passing backslash-based paths as stdin to xargs

Prelude System.Process> readCreateProcessWithExitCode (shell "xargs grep  -l 'Var'") "C:/Users/name/Projects/natLog/library/Types.hs"
(ExitSuccess,"C:/Users/name/Projects/natLog/library/Types.hs\n","")
Prelude System.Process> readCreateProcessWithExitCode (shell "xargs grep  -l 'Var'") "C:\\Users\\name\\Projects\\natLog\\library\\Types.hs"
(ExitFailure 123,"","grep: C:UsersnameProjectsnatLoglibraryTypes.hs: No such file or directory\n")

Something similar happens when running this in powershell, but weirdly not in cmd.exe .

This command is build when retrie tries to filter down a list of paths here.

This command eventually runs this function in System.Process so maybe the comment is related.

The quickest fix probably would be to substitute \ with / in the paths in the plugin.

[jneira]

@Tarmean wow, thanks for the further investigation, the fix could be convert the windows paths to unix style ones and does not seem to be too complex, would you be interested in make a pr?

[Tarmean]

The path conversion was really easy. Unfortunately the current System.Process implementation (well, hPutStr) also seems to mangle newlines in the stdin channel of the created process on windows by turning '\n' into '\r\n'. This also breaks xargs.

The stdin handle is created as 'intPtrToPtr (-1)' and I don't know enough about how this magic works to say how badly things would break if one were to call hSetNewlineMode.

An more palatable hack alternative might be to inline the first xargs invocation, i.e. turning

readCreateProcessWithExitCode (shell "xargs grep -l 'Fizz' | xargs grep -l 'Buzz'") paths

into

readCreateProcessWithExitCode (shell "grep -l 'Fizz' $paths | xargs grep -l 'Buzz'") ""

Either way I'm currently out of time so I will have to experiment more later.

[jneira]

Unfortunately the current System.Process implementation (well, hPutStr) also seems to mangle newlines in the stdin channel of the created process on windows by turning '\n' into '\r\n'.

I had to add a function to try to avoid that:

https://github.com/haskell/haskell-language-server/blob/2fef04193cfe2ac08372a008a5b5cfb6275d5bea/plugins/hls-hlint-plugin/src/Ide/Plugin/Hlint.hs#L437-L441

[Tarmean]

Replacing the usage of getTargetFiles with this version works on windows, the additions are filterChain0 and toUnixFilepath.

This is a gist and not a pullrequest to retrie because I noticed a couple of oddities and want to double check they aren't my fault. I think the weirdness are existing bugs in retrie, though

• Identifiers with ' aren't found because it generates searches like grep -L 'P''
• Operators are treated really weirdly. Might be because fixity isn't loaded? (Edit: Maybe only for builtins?)
• Retrie always demands multiple spaces so a rule foo [] doesn't fire if the occurrence is foo[], similar with superfluous brackets

Here is an example of weird operator behavior

{-# RULES "stepSum" forall x y. sum  (x : y) = x + (sum y) #-}
foo :: Int
foo = sum (1 : 2 : 3 : [])

foo :: Int
foo = 1 : 2 : 3 + (sum [])

[jneira]

I think this was closed by #2513, @Tarmean thanks!