Eager macro-expansion failure in lsp-terraform.el

[condy0919]

Thank you for the bug report

• [X] I am using the latest version of lsp-mode related packages.
• [X] I checked FAQ and Troubleshooting sections
• [X] You may also try reproduce the issue using clean environment using the following command: M-x lsp-start-plain

Bug description

According to lsp-protocol.el, the lsp interface of terraform should be:

(lsp-interface (terraform-ls:ModuleCalls (:v :module_calls) nil))
(lsp-interface (terraform-ls:Module (:name :docs_link :version :source_type :dependent_modules) nil))
(lsp-interface (terraform-ls:Providers (:v :provider_requirements :installed_providers) nil))

But three macro-expansion failures happened.

Module

Eager macro-expansion failure: (error "Unknown key: :docs-link.  Available keys: (:name :docs_link :version :source_type :dependent_modules)")

(lsp-defun construct-tf-module ((&terraform-ls:Module :name :docs-link :version :source-type :dependent-modules))
  "Construct `TF-MODULE' using MODULE."
  (make-tf-module :name name
                  :doc-link docs-link
                  :version version
                  :source-type source-type
                  :dependent-modules dependent-modules))

Providers

Eager macro-expansion failure: (error "Unknown key: :provider-requirements.  Available keys: (:v :provider_requirements :installed_providers)")

(lsp-defun lsp-terraform-ls--providers-to-tf-package ((&terraform-ls:Providers :provider-requirements :installed-providers))

ModuleCalls

Eager macro-expansion failure: (error "Unknown key: :module-calls.  Available keys: (:v :module_calls)")

(lsp-defun lsp-terraform-ls--modules-to-tf-module ((&terraform-ls:ModuleCalls :module-calls))
  "Convert MODULES-TREE-DATA to list of `TF-MODULE'."
  (let* ((modules (-map (lambda (x) (construct-tf-module x)) module-calls)))
    modules))

Steps to reproduce

Nothing

Expected behavior

No failures

Which Language Server did you use?

Nothing

OS

Linux

Error callstack

No response

Anything else?

No response

[yyoncho]

cc @psibi

[psibi]

@condy0919 Thanks for the bug report! I tried using lsp-plain-start with my configuration and I wasn't able to reproduce.

Is there any specific steps that I can follow to reproduce this issue ? Sharing your configuration might help me.

[condy0919]

@condy0919 Thanks for the bug report! I tried using lsp-plain-start with my configuration and I wasn't able to reproduce.

Is there any specific steps that I can follow to reproduce this issue ? Sharing your configuration might help me.

Enabling byte-compile (setq no-byte-compile nil) then open /tmp/a.cpp, there will be a failure message in *Messages* buffer.

https://github.com/emacs-lsp/lsp-mode/blob/master/scripts/lsp-start-plain.el#L37=

[psibi]

@condy0919 Thanks, I checked no-byte-compile and it's already set nil for me. I did what you mentioned and this is the error message I'm getting:

Company backend ’company-clang’ could not be initialized:
Company found no clang executable
Wrote /tmp/a.cpp

[condy0919]

@condy0919 Thanks, I checked no-byte-compile and it's already set nil for me. I did what you mentioned and this is the error message I'm getting:

Company backend ’company-clang’ could not be initialized:
Company found no clang executable
Wrote /tmp/a.cpp

Install clang or open an elisp file since lsp is in prog-mode-hook.

[psibi]

Install clang or open an elisp file since lsp is in prog-mode-hook.

Sure, I installed clangd and nothing happened. This is what I saw in my messages buffer:

Wrote /tmp/foo.cpp

I also opened an elisp file and I can't observe anything. I think it would be helpful if you can share your lsp-mode configuration that you are using.

[yyoncho]

@condy0919 are you reproducing the issue with M-x lsp-start-plain?

[condy0919]

@condy0919 are you reproducing the issue with M-x lsp-start-plain?

Yes, except for the nil no-byte-compile variable.

[condy0919]

I will then provide a minimal recipe init.el

[condy0919]

Sorry to disturb, I'm unable to reproduce after deleting all packages and reinstalling them again. :'(

The reason I guess is the *.elc files is generated by Emacs master at commit1, but later they're run on Emacs 29 at commint2.

[ZWindL]

Met the same issue today. Solution:

1. cd into your lsp-mode package dir
2. remove all .elc file
3. reboot emacs

[mattsawyer77]

I hit this issue today as well. I'm using emacs 29, and commit 0dfe2145734f91c79aa3bbe666327736982811d7 of lsp-mode.

The other workarounds posted above didn't seem to work for me, but I don't use terraform these days, so I just removed the client from the list:

(delete 'lsp-terraform lsp-client-packages)

Now all my lsp clients (excluding terraform, of course) seem to work normally.

Another user has forked lsp-mode to add this commit for a seemingly more correct fix: https://github.com/SteVwonder/lsp-mode/commit/db55e4f0700877d68ac4ef173ecbe12f4fbac710

Perhaps this issue should be reopened?

[shackra]

I'm experiencing this as well since last week when opening a Go file.

Error running timer: (error "Eager macro-expansion failure: (error \"Unknown key: :docs-link.  Available keys: (:name :docs_link :version :source_type :dependent_modules)\")")

[meiji163]

Got same error as @shackra

[emdash-ie]

EDIT: I don’t think this actually works – I’m not sure why it appeared to fix my problem, but the problem later came back and my changes were the cause this time, with the problem fixed by removing my changes.

Original comment:

I’ve been having this for a while now, and have fixed my own setup by making what I think is the opposite edit to the one made by SteVWonder: https://github.com/emdash-ie/lsp-mode/commit/c05484d10563862a1dd386170bdd601615cce149

I’m using emacs 29 on macOS via doom emacs, so I don’t have a minimal reproduction, but I wanted to describe what worked for me in case it’s helpful in diagnosing the issue:

as I made these edits in my local straight copy of lsp-mode, I saw the error message (which was appearing when I called 'revert-buffer in a non-terraform buffer for which lsp-mode was configured) move from reporting

Error running timer: (error "Eager macro-expansion failure: (error \"Unknown key: :docs-link. Available keys: (:name :docs_link :version :source_type :dependent_modules)\")")

to

Error running timer: (error "Eager macro-expansion failure: (error \"Unknown key: :source-type. Available keys: (:name :docs_link :version :source_type :dependent_modules)\")")

and so on.

Interestingly, flycheck is reporting the exact opposite message now, claiming that “:docs_link” is an unknown key and “:docs-link” should be available.

Sorry if this is unhelpful! The fact that it’s only started recently does make me think this is caused by something elsewhere, e.g. a byte-compilation issue, but I don’t know enough about that to be able to look into it properly.

[betaprior]

This is absolutely a problem that is impacting lots of users, as well as dozens of people in this thread alone. In one of the issues linked, @yyoncho claims a clean install fixes this, but for me this failure arose in a brand-new from-scratch Doom installation, so unless I'm not understanding what "clean install" means, I don't think this is true.

The only robust solution for Doom uses seems to be to use a forked version of lsp-mode with a patch like @emdash-ie posted above, which is really not ideal. I understand that this may be hard to definitively repro, but this breaks lots of people, and given that the proposed patches work, it's unfortunate that this issue is not being addressed.

[psibi]

I'm not understanding what "clean install" means, I don't think this is true.

I have seen others cleaning it up like this which made it work: rm -rf ~/.emacs.d

I understand that this may be hard to definitively repro, but this breaks lots of people, and given that the proposed patches work

IIUC, the proposed patch will break the functionality for terraform-ls users.

[kprav33n]

The other workarounds posted above didn't seem to work for me, but I don't use terraform these days, so I just removed the client from the list:

(delete 'lsp-terraform lsp-client-packages)

I started seeing this issue with Go files today with Doom on Emacs 29. I tried many things including purging elc files, reinstalling all modules, deleting lsp-mode repo from cache, etc. Nothing helped.

As I'm not using Terraform, I followed @mattsawyer77's suggestion and I was able to get everything working back again. I added the following section to ~/.doom.d/config.el.

(after! lsp-mode
  ;; https://github.com/emacs-lsp/lsp-mode/issues/3577#issuecomment-1709232622
  (delete 'lsp-terraform lsp-client-packages))

[numkem]

The answer from @kprav33n is working as expected but I do lsp-terraform myself so I'd like to find a "real" fix.

I tried deleting all *.elc files from .local/straight/build-2x.x/lsp-mode/*.elc as it was reported in the Doom Emacs Discord but that didn't do anything for me.

[Balooga]

The only robust solution for Doom uses seems to be to use a forked version of lsp-mode with a patch like @emdash-ie posted above, which is really not ideal. I understand that this may be hard to definitively repro, but this breaks lots of people, and given that the proposed patches work, it's unfortunate that this issue is not being addressed.

Doom user here, and Go mode is quite broken for me in Emacs 29. Reverting to Emacs 28 fixes the problem.

[JacobJanzen]

Came across the same issue today in Doom Emacs. I don't use terraform, so @kprav33n's solution works for me, but this issue really needs a real fix as staying on an old version of Emacs and removing functionality from the mode are not good long-term solutions.

[yyoncho]

I cannot think of a reason for that to happen. Is there anyone actually managing to reproduce it with M-x lsp-start-plain? FWIW We can rewrite the code in lsp-terraform.el to use lsp-get instead of using destructoring.

[psibi]

If someone wants to send a patch using lsp-get, I would be happy to review.

I don't use doom emacs, so probably I could be mis-understanding how things work there - but from a quick look it looks like they are pinning commits: https://github.com/doomemacs/doomemacs/blob/03d692f129633e3bf0bd100d91b3ebf3f77db6d1/modules/tools/lsp/packages.el#L12C28-L12C68

which seem to use lsp-mode from Nov 2023 which is quite old and may have this bug: https://github.com/emacs-lsp/lsp-mode/commit/d441f3d268a31a4a514af2db506a5eec47ee617d

[JacobJanzen]

This change seems to fix the problem for me. It looks like there was a typo in the name of the variable where a - is used instead of a _. It looks like it fails in the reverse way in the tests, so the variable name must have been fixed in a newer commit. I'll open an issue with Doom Emacs to update lsp-mode. Changing the pin to the latest version of lsp-mode doesn't fix the error.

[psibi]

@JacobJanzen Is that bug reproducible to you when you do lsp-start-plain ?

I'll open an issue with Doom Emacs to update lsp-mode. Changing the pin to the latest version of lsp-mode doesn't fix the error.

My guess is if someone is using older version of doom emacs which ships a older lsp-mode.

https://github.com/emacs-lsp/lsp-mode/pull/4309 change seems to fix the problem for me.

Unfortunately that fix will break terraform-ls functionality since we won't be passing the parameters as expected by the terraform-ls server.

[JacobJanzen]

@psibi The error does not seem to appear in the compile log with lsp-start-plain.

[emdash-ie]

My setup now works without error with my changes removed – lsp-mode is pinned to https://github.com/emacs-lsp/lsp-mode/commit/d441f3d268a31a4a514af2db506a5eec47ee617d and unmodified, and I can’t reproduce the error at all. (This also seems to have happened on a few other issues describing this problem.) Given this, I think @yyoncho and @psibi are correct that there is no bug in lsp-mode, and that the problem is about some odd byte-compilation clash. (Maybe running doom build fixes it?)

[emdash-ie]

My setup now works without error with my changes removed

This was premature: I don’t understand why it appeared to be fixed, but I still get this error. My current fix (which sometimes only lasts until I restart emacs, and sometimes confusingly lasts longer) is to evaluate these three lines using eval-region. (This feels like it should point towards what the problem is, but I don’t understand enough about how the code is normally being evaluted to be able to figure it out.)

[MatthewMSaucedo]

As I'm not using Terraform, I followed @mattsawyer77's suggestion and I was able to get everything working back again. I added the following section to ~/.doom.d/config.el.

(after! lsp-mode
  ;; https://github.com/emacs-lsp/lsp-mode/issues/3577#issuecomment-1709232622
  (delete 'lsp-terraform lsp-client-packages))

This worked for me too - same issue as others here; LSP not working (only for go, strangely? Python is fine).

[WJCFerguson]

This suddenly started happening for me too - all I did was update my OS packages (Ubuntu 20.04), nothing changed AFAIK in my Emacs config or packages. I tried updating all my packages including lsp in case it helped, then fully reloaded all my elpa packages from scratch, but nothing helped. Since I don't use terraform, https://github.com/SteVwonder/lsp-mode/commit/db55e4f0700877d68ac4ef173ecbe12f4fbac710 seemed to work for me.

[ryan-m-may]

This is happening to me as well, very infuriating, same symptoms, and the fix above doesn't actually to be solving the problem unfortunately.

Since I'm using doom emacs, I went ahead and just started trying out eglot, seems like the issue is solved. Edit: nope, it made a liar out of me.

[Balooga]

Since I'm using doom emacs, I went ahead and just started trying out eglot, seems like the issue is solved. Edit: nope, it made a liar out of me.

This worked for me in Doom Emacs

[Cbeck527]

I also ran into this issue with Doom Emacs, specifically when opening a go file. Since I also use Terraform regularly I didn't want to disable it if I didn't have to.

I ended up diving into my emacs config directory and brute-forced changes until the lsp server started successfully in my golang buffer. I don't know how sustainable this is, but I've been using it for the majority of day and haven't run into any issues. I'm almost certain this won't survive a doom package upgrade, so unless there's a more permanent solution I'm just going to keep running with this temp fix 😆

On my machine, lsp-mode is cloned to ~/.emacs.d/.local/straight/repos/lsp-mode

diff --git clients/lsp-terraform.el clients/lsp-terraform.el
index 686a5da61..b9bbbb3c7 100644
--- clients/lsp-terraform.el
+++ clients/lsp-terraform.el
@@ -302,7 +302,7 @@ This is a synchronous action."
                    :installed-version installed-version
                    :version-constraint (lsp-get provider :version_constraint)))

-(lsp-defun construct-tf-module ((&terraform-ls:Module :name :docs-link :version :source-type :dependent-modules))
+(lsp-defun construct-tf-module ((&terraform-ls:Module :name :docs_link :version :source_type :dependent_modules))
   "Construct `TF-MODULE' using MODULE."
   (make-tf-module :name name
                   :doc-link docs-link
@@ -310,7 +310,7 @@ This is a synchronous action."
                   :source-type source-type
                   :dependent-modules dependent-modules))

-(lsp-defun lsp-terraform-ls--providers-to-tf-package ((&terraform-ls:Providers :provider-requirements :installed-providers))
+(lsp-defun lsp-terraform-ls--providers-to-tf-package ((&terraform-ls:Providers :provider_requirements :installed_providers))
   "Convert PROVIDERS-TREE-DATA to list of `tf-package'."
   (let* ((provider-requirements-keys (hash-table-keys provider-requirements))
          (installed-versions (mapcar (lambda (x) (lsp-get installed-providers (make-symbol (format ":%s" x)))) provider-requirements-keys))
@@ -318,7 +318,7 @@ This is a synchronous action."
          (tf-packages (-zip-with (lambda (x y) (construct-tf-package x y)) providers installed-versions)))
     tf-packages))

-(lsp-defun lsp-terraform-ls--modules-to-tf-module ((&terraform-ls:ModuleCalls :module-calls))
+(lsp-defun lsp-terraform-ls--modules-to-tf-module ((&terraform-ls:ModuleCalls :module_calls))
   "Convert MODULES-TREE-DATA to list of `TF-MODULE'."
   (let* ((modules (-map (lambda (x) (construct-tf-module x)) module-calls)))
     modules))

[45mg]

Another Doom user here. I don't know why, but if I put (require 'lsp-mode) in my config.el it fixes the issue. Perhaps https://github.com/emacs-lsp/lsp-mode/issues/4222#issuecomment-1795696213 can provide some clues?

(for non-Doom users, the effect of loading it like that in config.el is that it's loaded on startup, not lazy-loaded)