Issues around th warning (`This HLS binary does not support Template Haskell`) and build hls with dynamic linking

jneira commented 2 years ago

After adding the warning about TH a bunch of issues have come up:

The instructions to build in the troubleshooting guide were incomplete, it did not include specific commands for stack or ghcup compile hls: i tried to fix that with #2647
Users with env where th was working fine has been compelled to build hls from source only to take ride of the warning (but well, we should have expected this i guess).
If you build hls with --enable-executable-dynamic (which links dynamically only the system libs) instead of --ghc-options="-dynamic" (which additionally links dynamically all haskell deps) the warning does not go away. However i thought the first option was sufficient to workaround th problems (maybe i am wrong but i infer that from comments like this)
As reported in irc, stack install haskell-language-server --ghc-options="dynamic" seems to not work out of the box. It seems you have to build it both statically and dynamically (i am trying to reproduce)
An user in macOS has got this error trying cabal install pkg:haskell-language-server --ghc-options="-dynamic":

src/Distribution/Extra/Doctest.hs:34:8: error:
    Could not find module ‘Prelude’
    Perhaps you haven't installed the "p_dyn" libraries for package ‘base-4.14.3.0’?
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
34 | module Distribution.Extra.Doctest (

jneira commented 2 years ago

As reported in irc, stack install haskell-language-server --ghc-options="dynamic" seems to not work out of the box. It seems you have to build it both statically and dynamically (i am trying to reproduce)

I've reproduced the error in wsl ubuntu with stack install haskell-language-server --ghc-options="-dynamic"

hiedb                             > [1 of 9] Compiling HieDb.Compat
hiedb                             >
hiedb                             > /tmp/stack-6de8f3e7cd60c56b/hiedb-0.4.1.0/src/HieDb/Compat.hs:84:1: error:
hiedb                             >     Bad interface file: /mnt/d/hls/.stack-work/install/x86_64-linux/c3ef80eb51e5bf52e59152c0adb03374e24f1447c1b9109c398861481f8ffd97/8.10.7/lib/x86_64-linux-ghc-8.10.7/hie-compat-0.2.1.1-6OvyfjjnZ4IJE2CvK3kpC3/Compat/HieTypes.hi
hiedb                             >         mismatched interface file ways (wanted "", got "dyn")
hiedb                             >    |
hiedb                             > 84 | import Compat.HieTypes
hiedb                             >    | ^^^^^^^^^^^^^^^^^^^^^^

jneira commented 2 years ago

If you build hls with --enable-executable-dynamic (which links dynamically only the system libs) instead of --ghc-options="-dynamic" (which additionally links dynamically all haskell deps) the warning does not go away. However i thought the first option was sufficient to workaround th problems (maybe i am wrong but i infer that from comments like this)

@pepeiborra please, could confirm if --enable-executable-dynamic would be sufficient to make th evaluation reliable as commented in https://github.com/haskell/haskell-language-server/issues/1982#issuecomment-873396136? In that case, would be possible change the alert condition to skip it in that case. Build with --enable-executable-dynamic is better imo as you dont have to rebuild all dependencies, only the executable and it seems it causes less problems (like the Perhaps you haven't installed the "p_dyn" libraries mentioned above).

pepeiborra commented 2 years ago

If you build hls with --enable-executable-dynamic (which links dynamically only the system libs) instead of --ghc-options="-dynamic" (which additionally links dynamically all haskell deps)

What is the source for this information? It doesn't match my understanding, which is:

System libs are always linked dynamically unless --enable-executable-static is used.
Haskell libs are always linked statically unless --enable-executable-dynamic is used.

the warning does not go away.

But it does in my system! (Mac OS). Have you been able to validate this information yourself?

In that case, would be possible change the alert condition to skip it in that case

The condition is accurate (I think). So if the condition is firing, I think the risk of segfaults still exists.

Perhaps you haven't installed the "p_dyn" libraries for package ‘base-4.14.3.0’?

This is what happens if you try to build both with --enable-executable-dynamic and with profiling at the same time. Unfortunately GHC doesn't ship with compilation artifacts for this mode, and thus it can never work. We can add this information in the HLS user's guide.

jneira commented 2 years ago

What is the source for this information? It doesn't match my understanding, which is:

Firstly an user in discord built hls with --ghc-options="-dynamic" and then with --enable-executable-dynamic and applied ldd to both executables, the first one had all haskell libs and the second one only the system libs. Also the first one caused a full recompilation of all hls dependencies.

Then i got to reproduce locally in wsl ubuntu. Will double check, in ci if possible to get a shareable reproducer

But it does in my system! (Mac OS). Have you been able to validate this information yourself?

Second hand, the same discord user in manjaro checked the first executable made the alert go away and the second one did not and another user in macos checked the alert continued being showed building with --enable-executable-dynamic (it could not check the other case cause they could not build hls with --ghc-options="-dynamic" due to the mentioned error)

Will check myself locally in wsl ubuntu

This is what happens if you try to build both with --enable-executable-dynamic and with profiling at the same time.

It might be, the user is a beginner who probably dont know even how to enable profiling (but maybe it is activated and they were not aware). That was trying to build hls itself to get the wanted dynamically linked executable after checking it out the project. We dont have profiling enabled in the project but maybe it is enabled in the global cabal config :thinking:

The entire discord talk is in https://discord.com/channels/280033776820813825/505370075402862594/937443543667056711 and posterior comments

jneira commented 2 years ago

What is the source for this information? It doesn't match my understanding, which is:
* System libs are always linked dynamically unless `--enable-executable-static` is used.

* Haskell libs are always linked statically unless `--enable-executable-dynamic` is used.

see also @hasufell comment about diff between --enable-executable-dynamic and --ghc-options="-dynamic" here: https://github.com/haskell/haskell-language-server/issues/1431#issuecomment-968854203

pepeiborra commented 2 years ago

What is the source for this information? It doesn't match my understanding, which is:

Firstly an user in discord built hls with --ghc-options="-dynamic" and then with --enable-executable-dynamic and applied ldd to both executables, the first one had all haskell libs and the second one only the system libs. Also the first one caused a full recompilation of all hls dependencies.

Probably user error. --enable-executable-dynamic definitely links the Haskell libs dynamically.

Then i got to reproduce locally in wsl ubuntu. Will double check, in ci if possible to get a shareable reproducer

But it does in my system! (Mac OS). Have you been able to validate this information yourself?

Second hand, the same discord user in manjaro checked the first executable made the alert go away and the second one did not and another user in macos checked the alert continued being showed building with --enable-executable-dynamic (it could not check the other case cause they could not build hls with --ghc-options="-dynamic" due to the mentioned error)

Building with --ghc-options="-dynamic" is not advisable, as it will try to build every dependency with -dynamic which is unnecessary. This is also what @hasufell was trying to point out.

hasufell commented 2 years ago

Building with --ghc-options="-dynamic" is not advisable

The other way around. --enable-executable-dynamic is broken with cabal v2-install (which ghcup uses), similar to --enable-executable-stripping. There are countless bug reports on cabal about it:

jneira commented 2 years ago

Building with --ghc-options="-dynamic" is not advisable, as it will try to build every dependency with -dynamic which is unnecessary. This is also what @hasufell was trying to point out.

That was also my understanding, however we are suggesting that (ghc-options variant) in the actual troubleshooting guide, precisely due to

The other way around. --enable-executable-dynamic is broken with cabal v2-install (which ghcup uses), similar to --enable-executable-stripping. There are countless bug reports on cabal about it:

And for stack it seems simply there is no --enable-executable-dynamic, although according to docs ghc-options only is applied to local targets by default

pepeiborra commented 2 years ago

I missed that change to the cabal instructions in the troubleshooting section. I don't think that cabal install --ghc-options=-dynamic works at all, so we should revert it and refresh the users guide.

If cabal install --enable-executable-dynamic doesn't work, then that's a Cabal bug and we can report it to the Cabal issue tracker. Has it been reported already @hasufell ?

hasufell commented 2 years ago

I missed that change to the cabal instructions in the troubleshooting section. I don't think that cabal install --ghc-options=-dynamic works at all, so we should revert it and refresh the users guide.

Why would it not work? I'm using that right now.

If cabal install --enable-executable-dynamic doesn't work, then that's a Cabal bug and we can report it to the Cabal issue tracker. Has it been reported already @hasufell ?

I linked two of them here https://github.com/haskell/haskell-language-server/issues/2659#issuecomment-1025637929

pepeiborra commented 2 years ago

I linked two of them here #2659 (comment)

I am confused: those tickets don't seem to mention --enable-executable-dynamic. So the problem is with cabal install then?

hasufell commented 2 years ago

I linked two of them here #2659 (comment)

I am confused: those tickets don't seem to mention --enable-executable-dynamic. So the problem is with cabal install then?

Those are all the same issue with similar flags.

jneira commented 2 years ago

I've built hls with --enable-executable-dynamic and ldd gives me only system libs, should not it list haskell libs?

jneira@FNTSY:/mnt/d/hls$ cabal install pkg:haskell-language-server --enable-executable-dynamic --installdir ~/.cabal/bin/esd
Wrote tarball sdist to
/mnt/d/hls/dist-newstyle/sdist/haskell-language-server-1.6.1.0.tar.gz
.....
Resolving dependencies...
Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - hls-plugin-api-1.3.0.0 (lib) (requires build)
.....
 - haskell-language-server-1.6.1.0 (exe:haskell-language-server-wrapper) (requires build)
 - haskell-language-server-1.6.1.0 (exe:haskell-language-server) (requires build)
Starting     hls-plugin-api-1.3.0.0 (lib)
..............
Installing   haskell-language-server-1.6.1.0 (exe:haskell-language-server-wrapper)
Completed    haskell-language-server-1.6.1.0 (exe:haskell-language-server-wrapper)
Installing   haskell-language-server-1.6.1.0 (exe:haskell-language-server)
Completed    haskell-language-server-1.6.1.0 (exe:haskell-language-server)
Symlinking 'haskell-language-server' to
'/home/jneira/.cabal/bin/esd/haskell-language-server'
Symlinking 'haskell-language-server-wrapper' to
'/home/jneira/.cabal/bin/esd/haskell-language-server-wrapper'
jneira@FNTSY:/mnt/d/hls$ ldd /home/jneira/.cabal/bin/esd/haskell-language-server
        linux-vdso.so.1 (0x00007fffc2f26000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4e61e99000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f4e61c7f000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f4e61a55000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f4e6184d000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f4e6164a000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4e61446000)
        libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f4e611c3000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4e60e24000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4e60b20000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f4e620b6000)

maybe it is due the mentioned bug?

pepeiborra commented 2 years ago

All right, I can confirm that cabal install --enable-executable-dynamic doesn't work as it doesn't produce a binary with the dynamic RTS. @hasufell is right.

I think it's time to bite the bullet and add the dynamic flag in the haskell-language-server.cabal descriptor

jneira commented 2 years ago

Hmm i've revised almost all issues about th and compiler crashes and we suggested --enable-executable-dynamic many times. And i dont remember anyone reporting it did not work. :thinking:

pepeiborra commented 2 years ago

cabal build --enable-executable-dynamic works fine

jneira commented 2 years ago

Nice to see thing are slowly becoming clearer, quoting myself from the pr #2668:

otoh at this point I am not sure if this is needed for sure to get a hls binary which works for TH

are we 100% sure a binary dinamically linked in the target os only against system libs (the default link mode in Linux!) does not work? maybe simply build hls in the target os is the key, as we are compiling exactly with same ghc used in the projects

the fact we did not get to reproduce the th issue in CI makes very difficult get a solution

so I think it worths to investigate a little bit more the issue before enable this which could open a new whole kind of issues

istathar commented 2 years ago

Hey. Thanks for replying to my messages on Twitter.

Several people on my team and I have gone multiple rounds on this today. I'll try my best to summarize what we've found:

This all started because of the warning about "does not support Template Haskell". Ok
Advice was to build locally, with stack install haskell-language-server --ghc-options=-dynamic. Ok, sure! [I still have my open question about how the -wrapper thing works and how we're supposed to name binaries in ~/.local/bin but minor detail). I've run into problems with -dynamic on Linux before; it works with a hello Hello World project, but not for something more complex like pandoc. Hm
I was able to build hls locally that way!
@dmvianna was not!
I tried a fresh build on an identical Fedora Linux build server (one of the ones we use for our GitHub actions, actually) and it failed as it did for Daniel :(. So what the heck was I doing differently?
The build failure is the mismatched interface file ways (wanted "", got "dyn") down in hiedb for example.
maybe I had snuck in a build without -dynamic first? Maybe that's why it worked for me?
In any case when it was working for me earlier I had no VS Code warning and no segfaults. I still don't know how I successfully built it.
I rebuilt locally without -dynamic. The warning is back. The segfaults are NOT.

This is all from HEAD of this Git repo.

So, yeay?

My guess is that the fully static build that your release pipeline is making (on Alpine? Nix? whatever) and making available via GitHub (?) to VS Code is not working because glibc + static = boom, but that building locally brought me back to the Haskell default on Linux of dynamic executable but with the Haskell code statically linked into it.

[it sure would be nice if we could actually get dynamic linking of the Haskell code working reliably on Linux, but that's a conversation for another day!]

So I think my report is that the other work you have been doing recently to impove the Template Haskell situation has paid off, but that the problem with the release binaries remains (and isn't likely to go away any time soon).

What line do I hack out to make the warning go away? :)

AfC

jneira commented 2 years ago

@istathar thanks for the feedback and sorry for the trouble caused As commented in the issue we are reviewing what could be the best way to get the correct hls binary given the limitations and bugs of our build tools, cabal and stack. There is an alternative in #2668 which also updates instructions so maybe you could give it a try in the envs where --ghc-options=-dynamic failed

jneira commented 2 years ago

cabal build --enable-executable-dynamic works fine

I've just checked it in debian 10 and i observed:

the final executable has all all haskell dynamically linked
dependencies seems to be compiled with the ghc option -dynamic-too, you can check in the build it is generating bot .o and .dyn_o files
if you do cabal build --ghc-options="-dynamic" only .dyn_o files are generated (ant do it after the previous one trigger a full rebuild)
if you do a raw cabal build only .o files are generated
if you intercale them in whatever order you get always a full rebuild (although -dynamic-too includes both versions)
- so if you have hls built no dynamic at all, the default, cabal build --enable-executable-dynamic causes also a full rebuild

so to avoid errors like the posted above:

Bad interface file: /mnt/d/hls/.stack-work/install/x86_64-linux/c3ef80eb51e5bf52e59152c0adb03374e24f1447c1b9109c398861481f8ffd97/8.10.7/lib/x86_64-linux-ghc-8.10.7/hie-compat-0.2.1.1-6OvyfjjnZ4IJE2CvK3kpC3/Compat/HieTypes.hi
hiedb                             >         mismatched interface file ways (wanted "", got "dyn")

maybe we should consider use -dynamic-too instead -dynamic

istathar commented 2 years ago

I just did

$ stack install --stack-yaml=stack-8.10.7.yaml --ghc-options="-dynamic-too"

and it appears to have worked, both remotely on a build server and locally, tested in VS Code. To be honest, I don't know how much of a clean-room I have at this point but it seems to NOT be segfaulting ... but it still DOES have the annoying warning at Code startup.

jneira commented 2 years ago

To be honest, I don't know how much of a clean-room I have at this point but it seems to NOT be segfaulting ... but it still DOES have the annoying warning at Code startup.

ok, thanks for trying, what about #2668?, to have a confirmation it works in your env with stack would be helpful

it could be done with git checkout dynamic && stack install haskell-language-server (using the default stack.yaml which uses last lts/ghc-8.10.7)

istathar commented 2 years ago

(sorry, I should clarify, two changes: 1. lts-18.23 and 2. removed the $everything clause)

Happy to try that branch. Will do so in the morning.

jneira commented 2 years ago

(sorry, I should clarify, two changes: 1. lts-18.23 and 2. removed the $everything clause)

oh yeah, we have a little bit outdated the lts, the -haddock is needed to get doc on hover on hls itself so it only would be handy if you are going to hack hls itself

istathar commented 2 years ago

So I tried it.

$ git checkout dynamic
$ stack install

ldd reports a fully dynamic executable (!), not just the system libraries.

Completed (5 files worked, 0 files failed)

which not only is not a segfault, but is also better than the "0 worked 5 failed" from before.

No warning from VS Code on startup.

Summary: your 'dynamic' branch works with both stack.yaml and stack-8.10.7.yaml. So does 'master' compiled with --ghc-options=-dynamic-too.

jneira commented 2 years ago

ldd reports a fully dynamic executable (!), not just the system libraries.

yeah that was expected

which not only is not a segfault, but is also better than the "0 worked 5 failed" from before. No warning from VS Code on startup.

thanks! happy to see it is working

out of curiosity, did you have actually segfaults due to the use of TH?

jneira commented 2 years ago

maybe we should consider use -dynamic-too instead -dynamic

nvm, the regular build (without any linking option) also builds the libs with -dynamic-too, and after it building #2668 did not rebuild deps

and it does not make the executable be dynamically linked as checked in https://github.com/haskell/haskell-language-server/issues/2659#issuecomment-1026747361

istathar commented 2 years ago

out of curiosity, did you have actually segfaults due to the use of TH?

Yes. I was the one reopening https://github.com/haskell/haskell-language-server/issues/800#issuecomment-1016008848

dmvianna commented 2 years ago

$ git checkout dynamic $ stack install

Works for me as well! 🥳

Anrock commented 2 years ago

Another minor issue: the instructions link leads to https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries instead of https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries-and-template-haskell-support.

jneira commented 2 years ago

Another minor issue: the instructions link leads to https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries instead of https://haskell-language-server.readthedocs.io/en/latest/troubleshooting.html#static-binaries-and-template-haskell-support.

my bad (changed the title to make it more discoverable), will correct it asap. To have hardcoded urls in the code is a real PITA

dmvianna commented 2 years ago

Final question: how do you compile for multiple GHC versions? How do you make haskell-language-server-wrapper use the right binary? Just by naming it correctly? Is this manual or by script?

hasufell commented 2 years ago

Final question: how do you compile for multiple GHC versions?

If you compile via ghcup, this is done for you:

ghcup compile hls -v 1.6.1.0 -j10 --ghc 8.10.7                                      --cabal-project-local https://paste.tomsmeding.com/eKoT0X8q/raw/1
ghcup compile hls -v 1.6.1.0 -j10 --ghc 9.0.2  --cabal-project cabal-ghc90.project  --cabal-project-local https://paste.tomsmeding.com/eKoT0X8q/raw/1
ghcup compile hls -v 1.6.1.0 -j10 --ghc 9.2.1  --cabal-project cabal-ghc921.project --cabal-project-local https://paste.tomsmeding.com/eKoT0X8q/raw/1

istathar commented 2 years ago

Is there a naming convention for the actual binaries that haskell-language-server-wrapper is expecting?

jneira commented 2 years ago

Is there a naming convention for the actual binaries that haskell-language-server-wrapper is expecting?

haskell-language-server-${ghcVersion}

https://haskell-language-server.readthedocs.io/en/latest/installation.html

We have to update install from source instructions

istathar commented 2 years ago

Ok, that's neat. fyi the VSCode plugin seems to place binaries with this naming pattern:

haskell-language-server-1.5.1-linux-8.10.7

burkettd commented 2 years ago

I ran ghcup compile hls -v 1.6.1.0 --ghc 8.10.7 and HLS is working in modules that use TemplateHaskell (Yay! :tada:). However I still get the warning "This HLS binary does not support Template Haskell." from coc.nvim. It's only a very minor inconvenience to close the warning when starting Neovim, but is there a setting I can change somewhere to suppress this warning?

hasufell commented 2 years ago

I ran ghcup compile hls -v 1.6.1.0 --ghc 8.10.7 and HLS is working in modules that use TemplateHaskell (Yay! tada). However I still get the warning "This HLS binary does not support Template Haskell." from coc.nvim. It's only a very minor inconvenience to close the warning when starting Neovim, but is there a setting I can change somewhere to suppress this warning?

Try

ghcup compile hls -v 1.6.1.0 --ghc 8.10.7 --cabal-project-local https://gist.githubusercontent.com/hasufell/8409effe97d33c36369e4d9906556c1e/raw/eb5f592b8569dd4ad9e7ea072f32d9dc4594ed83/gistfile1.txt

burkettd commented 2 years ago

ghcup compile hls -v 1.6.1.0 --ghc 8.10.7 --cabal-project-local https://gist.githubusercontent.com/hasufell/8409effe97d33c36369e4d9906556c1e/raw/eb5f592b8569dd4ad9e7ea072f32d9dc4594ed83/gistfile1.txt

That worked! Thanks a lot, @hasufell

tillydray commented 2 years ago

in case it's helpful, I tried ghcup compile hls -v 1.6.1.0 --ghc 8.10.7 but I was still getting errors and hls crashing. I also did ghcup nuke and installed stack, cabal, ghc, and hls via brew and that also didn't work.

The only thing that worked was git clone git@github.com:haskell/haskell-language-server.git && cd haskell-language-server && stack install haskell-language-server --stack-yaml=stack-8.10.7.yaml. Now I don't get errors or warning in emacs with lsp-mode, or nvim with neovim/nvim-lspconfig

haskell / haskell-language-server

Issues around th warning (`This HLS binary does not support Template Haskell`) and build hls with dynamic linking #2659