PR 135 Recent change in grammar causes issues in font locking (syntax highlighting)

[dhanak]

Hi,

I've been using Emacs 29.4 with julia-ts-mode for a while with great statisfaction. However, a recent update to the libtree-sitter-julia.so grammar causes the following error in font locking, and as a result, I don't get any syntax highlighting at all:

Error during redisplay: (jit-lock-function 1501) signaled (treesit-query-error "Node type error at" 177 "(assignment (identifier) @font-lock-variable-name-face (_)) (assignment (field_expression (identifier) \".\" (identifier) @font-lock-variable-name-face) (operator)) (assignment (bare_tuple (identifier) @font-lock-variable-name-face)) (assignment (bare_tuple (field_expression (identifier) \".\" (identifier) @font-lock-variable-name-face)) (operator)) (local_statement (identifier) @font-lock-variable-name-face) (let_binding (identifier) @font-lock-variable-name-face) (global_statement (identifier) @font-lock-variable-name-face)" "Debug the query with `treesit-query-validate'")

I assume something changed in the grammar that the ts-mode implementation didn't follow, but I can't figure out how to debug this. I see in the issue history that there have been similar issues in the past, so I'm hoping that there should be a fix for this.

I observed this issue first when I tried to update the grammar on April 12 this year, but I don't know exactly what version I had before that worked. The issue persists with the most recent grammar version.

Any help would be appreciated!

[dhanak]

Aha, I think I figured it out after all! bare_tuple has been renamed to open_tuple in this change, so the query for assignment in julia-ts-mode.el#L165 and two lines below bust be adjusted.

[dhanak]

Unfortunately, there are other changes to the grammar as well, e.g., no more short_function_definition, the name: tags are no longer valid, etc.

[dhanak]

This PR is responsible for these changes: https://github.com/tree-sitter/tree-sitter-julia/pull/135

Could this be the solution? Is it possible that these are all the necessary changes? Did I miss something? I get no errors now, but I can't be certain...

--- julia-ts-mode.el.orig   2024-08-29 12:57:10.007777566 +0200
+++ julia-ts-mode.el    2024-08-29 13:26:48.256695303 +0200
@@ -160,12 +160,14 @@
    :feature 'assignment
    `((assignment (identifier) @font-lock-variable-name-face (_))
      (assignment
-      (field_expression (identifier) "." (identifier) @font-lock-variable-name-face)
+      (field_expression
+       value: (identifier) "." (identifier) @font-lock-variable-name-face)
       (operator))
-     (assignment (bare_tuple (identifier) @font-lock-variable-name-face))
+     (assignment (open_tuple (identifier) @font-lock-variable-name-face))
      (assignment
-      (bare_tuple
-       (field_expression (identifier) "." (identifier) @font-lock-variable-name-face))
+      (open_tuple
+       (field_expression
+        value: (identifier) "." (identifier) @font-lock-variable-name-face))
       (operator))
      (local_statement (identifier) @font-lock-variable-name-face)
      (let_binding (identifier) @font-lock-variable-name-face)
@@ -187,15 +191,31 @@
    :language 'julia
    :feature 'definition
    `((function_definition
-      name: (identifier) @font-lock-function-name-face)
+      (signature (identifier) @font-lock-function-name-face))
      (function_definition
-      name: (field_expression (identifier) "." (identifier) @font-lock-function-name-face))
+      (signature
+       (call_expression (identifier) @font-lock-function-name-face)))
+     (function_definition
+      (signature
+       (call_expression
+        (field_expression
+         value: (identifier) "." (identifier) @font-lock-function-name-face))))
+     (macro_definition
+      (signature
+       (call_expression (identifier) @font-lock-function-name-face)))
      (macro_definition
-      name: (identifier) @font-lock-function-name-face)
-     (short_function_definition
-      name: (identifier) @font-lock-function-name-face)
-     (short_function_definition
-      name: (field_expression (identifier) "." (identifier) @font-lock-function-name-face)))
+      (signature
+       (call_expression
+        (field_expression
+         value: (identifier) "." (identifier) @font-lock-function-name-face))))
+     (assignment
+      (call_expression (identifier) @font-lock-function-name-face)
+      (operator))
+     (assignment
+      (call_expression
+       (field_expression
+        value: (identifier) "." (identifier) @font-lock-function-name-face))
+      (operator)))

    :language 'julia
    :feature 'error
@@ -261,8 +281,7 @@
    :override t
    `((type_clause (operator) (_) @font-lock-type-face)
      (typed_expression (_) "::" (_) @font-lock-type-face)
-     (typed_parameter
-      type: (_) @font-lock-type-face)
+     (unary_typed_expression (_) @font-lock-type-face)
      (where_clause "where"
                    (curly_expression "{"
                                      (binary_expression (identifier)
@@ -361,7 +380,6 @@
   (pcase (treesit-node-type node)
     ((or "abstract_definition"
          "function_definition"
-         "short_function_definition"
          "struct_definition")
      (treesit-node-text
       (treesit-node-child-by-field-name node "name")
@@ -401,7 +419,7 @@

   ;; Imenu.
   (setq-local treesit-simple-imenu-settings
-              `(("Function" "\\`function_definition\\|short_function_definition\\'" nil nil)
+              `(("Function" "\\`function_definition\\'" nil nil)
                 ("Struct" "\\`struct_definition\\'" nil nil)))

   ;; Fontification

[johannesnauta]

I am encountering the exact same issue:

Error during redisplay: (jit-lock-function 1) signaled (treesit-query-error "Node type error at" 177 "(assignment (identifier) @font-lock-variable-name-face (_)) (assignment (field_expression (identifier) \".\" (identifier) @font-lock-variable-name-face) (operator)) (assignment (bare_tuple (identifier) @font-lock-variable-name-face)) (assignment (bare_tuple (field_expression (identifier) \".\" (identifier) @font-lock-variable-name-face)) (operator)) (local_statement (identifier) @font-lock-variable-name-face) (let_binding (identifier) @font-lock-variable-name-face) (global_statement (identifier) @font-lock-variable-name-face)" "Debug the query with `treesit-query-validate'")
Error during redisplay: (jit-lock-function 1501) signaled (treesit-query-error "Node type error at" 177 "(assignment (identifier) @font-lock-variable-name-face (_)) (assignment (field_expression (identifier) \".\" (identifier) @font-lock-variable-name-face) (operator)) (assignment (bare_tuple (identifier) @font-lock-variable-name-face)) (assignment (bare_tuple (field_expression (identifier) \".\" (identifier) @font-lock-variable-name-face)) (operator)) (local_statement (identifier) @font-lock-variable-name-face) (let_binding (identifier) @font-lock-variable-name-face) (global_statement (identifier) @font-lock-variable-name-face)" "Debug the query with `treesit-query-validate'")

What is the easiest way to go back to a previous grammar, assuming that that one does work (like you mentioned)?

[johannesnauta]

Seems that even going back to a previous version (see below), v0.22.6 does not solve this specific issue.

To compile v0.22.6, I just cloned the repository and checked out 5df0420. Then, after installing tree-sitter-cli, I did from the tree-sitter-julia directory

$ tree-sitter generate
$ tree-sitter test  # seems good
$ gcc -shared -o libtree-sitter-julia.so src/parser.c src/scanner.c

[johannesnauta]

Sorry for the chain of comments, but going all the way back to before https://github.com/tree-sitter/tree-sitter-julia/pull/135 seems to do the trick. I am now at 0c088d1ad270f02c4e84189247ac7001e86fe342.

[dhanak]

...or, you can start using my fork of julia-ts-mode from here: https://github.com/dhanak/julia-ts-mode which fixes all (? or most) of the issues with the changed grammar.

[dhanak]

This is probably a duplicate of #21

[alecStewart1]

I'm in a similar boat as this issue and #21. I'm on Emacs 31 though.

It's not great, but you can have certain code blocks be evaluated and compiled based on Emacs version.

(when (>= emacs-major-version 30)
    (eval-and-compile 
       ;; your code here
     ))

Unfortunately I don't know if this package's repository will be updated in the near future. dhanak's might be worth switching to for now.

There's also treesit-auto. With it, you can use the regular julia-mode, and this package can remap that mode to a created julia-ts-mode:

https://github.com/renzmann/treesit-auto?tab=readme-ov-file#what-this-package-does

So there are options until things are updated.