Externally compiled tree sitter grammars do not work with the snap

[gernotk]

Hello!

Before starting to complain let me say thank you for your snap which really simplified setting up a modern Emacs environment for me!

I have been using Emacs' tree sitter integration for some time already e.g. when developing Typescript and JavaScript code -- but never had given it much deeper thought. To get the new "js-ts-mode" and "tsx-ts-mode" working I just went over there, compiled my tree sitter grammar of choice, copied the result into my .emacs.d/tree-sitter directory and went on hacking. Until the day I needed to reach for "yaml-ts-mode". I compiled the corresponding shared dynamic library on my Ubuntu 22.04 machine, copied the resulting "libtree-sitter-yaml.so" file again into .emacs.d/tree-sitter, tried to activate the mode and still hit the error claiming the language grammar was not available. Well, after a bit of debugging I realized this specific grammar module required C++ and it turns out Ubuntu 22.04's standard C++ library is incompatible with "core20" your Emacs' snap is based on:

listdc++.so.6: error: version lookup error: version 'GLIBCXX_3.4.29' not found (required by ~/.emacs.d/tree-sitter/libtree-sitter-yaml.so)

Workaround is: Install "g++-9" and re-compile.

But while I could just "cross-compile" tree sitter modules using a Ubuntu 20 compatible environment, in the case either you would upgrade your snap to core22 or me upgrade to Ubuntu 24, the module would not load again. So, I guess, the most convenient solution regarding tree-sitter grammar modules would possibly, ultimately be incorporating all of them inside your snap.

To do that snapcraft would need to fetch this repo here, batch compile all languages therein and copy the resulting shared libraries to some nice place along the link path inside your Emacs snap -- whenever you issue a new version of this snap.

Wouldn't that be good idea?

[alexmurray]

Regarding the initial issue - this is very similar to issues like https://github.com/alexmurray/emacs-snap/issues/71 and https://github.com/alexmurray/emacs-snap/issues/66 and is an unfortunate side-effect of trying to have a single distribution of emacs targeted at various different platforms with different glibc versions etc.

Then regarding the idea of bundling the tree sitter modules - this is definitely a good suggestion. On one hand this will make the emacs snap a bit larger (but likely not by a lot) but it will make it work better in more situations out-of-the-box. But on the other hand it would mean that these modules may not get updates very often so may not include bug fixes etc.

But overall I think this is a great suggestion. I'll see if I can find some time to try and implement it.

[alexmurray]

Note that I already patch treesit.el so that if you use it to compile the modules directly in emacs then they should work https://github.com/alexmurray/emacs-snap/blob/master/treesit.patch

But as you've seen, if you've compiled modules externally then they won't be expected to work.

[gernotk]

That's a good point. I tried running "treesit-install-language-grammar" many times and the only time it succeeded for me was for "JavaScript". All the other times Emacs would ask me "There is no recipe for X, do you want to build it interactively?" and I hit "y" but then end up with no errors but also no grammar modules. Perhaps I made mistake back then. However, even if it worked perfectly well, the compiled grammar modules would end up on the host system inside "~/.emacs.d/tree-sitter" and would break if the snap moved on to a higher core version.

[vzaliva]

I also could not make it work. I have externally compiled grammar which compiles and work from command line. For example I can do tree-sitter highlight file.core. However in emacs it never works. I tried:

 (use-package treesit-auto
   :config
   (global-treesit-auto-mode)
   (setq core-tsauto-config
         (make-treesit-auto-recipe
          :lang 'core
          :ts-mode 'core-ts-mode
          :url "https://github.com/vzaliva/tree-sitter-core"
          :revision "main"
          :source-dir "src"  
          :ext "\\.core\\'"))
   (add-to-list 'treesit-auto-recipe-list core-tsauto-config)
   (setq treesit-auto-langs '(core))
   (treesit-auto-add-to-auto-mode-alist '(core))
   )

and

 (use-package treesit-auto
   :ensure t
   :config
   (setq treesit-language-source-alist
         '((core . ("https://github.com/vzaliva/tree-sitter-core" "main" "src"))))
   (dolist (source treesit-language-source-alist)
     (unless (treesit-ready-p (car source))
       (treesit-install-language-grammar (car source))))
   (setq treesit-auto-langs '(core))
   (global-treesit-auto-mode)
   (add-to-list 'auto-mode-alist '("\\.core\\'" . core-ts-mode))
   )

is this because of this issue or something else? I am using emacs 29.1 on Ubuntu.

[alexmurray]

Yes it is a known issue that if you externally compile tree-sitter grammars then they will not work with the emacs snap as it uses a different libc etc - instead you will need to recompile them using the gcc etc as shipped within the emacs snap itself as detailed earlier in this issue.

[alexmurray]

But all you should need to do is delete any existing .so files and recompile them using as you have defined above via treesit-install-language-grammar and it should just do the right thing.

[vzaliva]

In the setup above, emacs is downloading source code from git repo and compiling it. I see in emacs log file:

Package ‘treesit-auto’ installed.
Cloning repository
Compiling library
Library installed to ~/.emacs.d/tree-sitter/libtree-sitter-core.so

$ file  ~/.emacs.d/tree-sitter/libtree-sitter-core.so
/home/lord/.emacs.d/tree-sitter/libtree-sitter-core.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=997063eb51973e332c670c824a334bab03816157, not stripped

When I try to open file.core I get "File mode specification error: (void-function core-ts-mode)"

[alexmurray]

Ah ok - perhaps this is an issue in treesit-auto - let me take a look.

[vzaliva]

OK, I was able to install grammar which treesit recognizes. The trick is to specify compiler as follows:

   (setq core-tsauto-config
         (make-treesit-auto-recipe
          :lang 'core
          :ts-mode 'core-ts-mode
          :url "https://github.com/vzaliva/tree-sitter-core"
          :revision "main"
          :source-dir "src"  
          :ext "\\.core\\'"
          :cc "/snap/emacs/2414/usr/bin/gcc-10"
          :c++ "/snap/emacs/2414/usr/bin/g++-10"
          ))

Hence I am considering this issue closed.

[alexmurray]

Hmm it is good you got it working but ideally you wouldn't have to specify either :cc or :c++ - from what I can see, if neither is specified it should specify nil for both and then the patched version of treesit.el within the emacs snap should use the compiler from within the emacs snap.

Do you by any chance have a local copy of treesit.el installed which may be overriding the one from within the emacs snap? Can you try running the following and let me know what it returns:

M-: (symbol-file 'treesit--install-language-grammar-1)

(this should show "/snap/emacs/current/usr/share/emacs/29.2/lisp/treesit.elc" within the minibuffer and the Messages buffer but I wonder if you see something different).

Also one quick thing - the path /snap/emacs/2414 will change each time a new revision of the snap is released, so it is better to specify it as /snap/emacs/current which will should always point to the currently installed revision.

[vzaliva]

Your command outputs /snap/emacs/current/usr/share/emacs/29.2/lisp/treesit.elc. It might have worked without it after the recent snap upgrade. I've initially had this problem with 29.1.

[alexmurray]

Ok, thanks for checking. Since this is still a valid issue but which we can't easily solve, I will leave it open so that others can find it if they are having similar troubles. Thanks.